flutter_native_vision_camera 0.1.0

High-performance Flutter FFI camera plugin with zero-copy preview textures, integrated MLKit/Vision barcode scanning, and low-latency native frame access for real-time on-device vision.

Flutter Native Vision Camera

A high-performance, FFI-powered camera plugin for Flutter, built for real-time on-device vision: direct, low-overhead access to camera frames for your own ML/CV code, integrated barcode/QR scanning, and full hardware control.

Status: Active development (0.0.x) — the API may change before 1.0.0. Device-verified on Android (Pixel 8) and iOS 18: preview, photo, video recording, barcode/QR scanning, zoom/torch/focus, orientation, and mirroring.

Demo

Preview + orientation	Barcode / QR scanning	FFI frame processor

Why use this instead of camera?

The official camera package is excellent for general capture. This package targets the niche it doesn't cover: a real-time frame pipeline where you read raw camera buffers for your own vision code, with integrated scanning, all sharing one GPU-texture preview.

Feature	camera (standard)	flutter_native_vision_camera
Preview	Platform texture	GPU texture (Android zero-copy SurfaceProducer; iOS CVPixelBuffer)
Frame access	startImageStream over the platform channel (serialized)	Direct FFI pointer access to plane buffers — no channel serialization
Native frame hook	—	Synchronous C/C++ plugin on the camera thread (zero-latency)
Barcodes / QR	Separate plugin	Integrated (Android MLKit, iOS Vision)

Features

• GPU-texture preview with automatic orientation.
• FFI frame access — read raw Y/U/V (Android) or BGRA (iOS) planes directly.
• Synchronous native C/C++ plugins invoked on the camera thread.
• Integrated barcode/QR scanning (Android MLKit, iOS Vision).
• Hardware controls — zoom, torch, exposure, tap-to-focus, video recording.

Requirements

• Flutter ≥ 3.22, Dart ≥ 3.8
• iOS 13.0+, Android minSdk 21+
• A physical device — simulators/emulators have no real camera.

Getting Started

Install

flutter pub add flutter_native_vision_camera

Permissions

iOS — add to ios/Runner/Info.plist:

<key>NSCameraUsageDescription</key>
<string>This app needs camera access to capture photos and video.</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app needs microphone access to record video with audio.</string>

Android — CAMERA and RECORD_AUDIO are declared by the plugin; request them at runtime (shown below).

Basic usage — preview

import 'package:flutter/material.dart';
import 'package:flutter_native_vision_camera/flutter_native_vision_camera.dart';

// Once, at startup (e.g. in main()):
initializeVisionCamera();

class CameraScreen extends StatefulWidget {
  const CameraScreen({super.key});
  @override
  State<CameraScreen> createState() => _CameraScreenState();
}

class _CameraScreenState extends State<CameraScreen> {
  final controller = CameraController();

  @override
  void initState() {
    super.initState();
    _start();
  }

  Future<void> _start() async {
    // 0. Permission (otherwise you get a black preview).
    if (await CameraPermissions.requestCameraPermission() !=
        PermissionStatus.granted) {
      return;
    }
    // 1. Pick a device.
    final devices = await CameraDevices.getAvailableCameraDevices();
    final back =
        CameraDevices.getCameraDevice(devices, CameraPosition.back) ??
        devices.first;
    // 2. Initialize + start.
    await controller.initialize(back, enablePhoto: true);
    await controller.setActive(true);
    if (mounted) setState(() {});
  }

  @override
  void dispose() {
    controller.dispose(); // Release the hardware.
    super.dispose();
  }

  @override
  Widget build(BuildContext context) =>
      CameraPreview(controller: controller, resizeMode: ResizeMode.cover);
}

Take a photo

takePhoto() requires enablePhoto: true at init.

final photo = await controller.takePhoto();
Image.file(File(photo.path)); // photo.width / photo.height / photo.path

Record video

Initialize with enableVideo: true, then:

final dir = await getTemporaryDirectory();
await controller.startRecording('${dir.path}/clip.mp4');
// ... seamlessly zoom / toggle torch while recording ...
await controller.stopRecording();

The mirror flag on initialize controls the front-camera selfie mirror for both the preview and the saved photo/video (default true; set false to save what the camera actually sees).

Scan barcodes / QR codes

await controller.initialize(
  device,
  codeScanner: CodeScannerConfiguration(
    codeTypes: [CodeType.qr, CodeType.ean13, CodeType.code128],
  ),
);
controller.onCodeScanned.listen((codes) {
  for (final code in codes) debugPrint('${code.type}: ${code.value}');
});

Lifecycle management

Release the hardware on dispose(), and pause/resume with the app lifecycle:

class _S extends State<MyCam> with WidgetsBindingObserver {
  @override
  void initState() { super.initState(); WidgetsBinding.instance.addObserver(this); }
  @override
  void dispose() {
    WidgetsBinding.instance.removeObserver(this);
    controller.dispose();
    super.dispose();
  }
  @override
  void didChangeAppLifecycleState(AppLifecycleState state) {
    if (!controller.isInitialized) return;
    if (state == AppLifecycleState.inactive ||
        state == AppLifecycleState.paused) {
      controller.setActive(false);
    } else if (state == AppLifecycleState.resumed) {
      controller.setActive(true);
    }
  }
}

Frame Processors

Run code for every camera frame — the package's headline feature.

Threading model — read this

Three options, lightest to heaviest:

• setFrameProcessor — a closure delivered asynchronously on the main isolate (via dart:ffi NativeCallable.listener). Easiest; keep the work light (FPS, brightness) or it competes with your UI.
• setFrameWorklet — a top-level function that runs on a background isolate for heavy work (ML/CV), with the main thread untouched. Results stream back via frameResults. See below.
• Native C/C++ hook — runs synchronously on the camera thread with zero added latency, for the very heaviest work.

await controller.setFrameProcessor((frame) {
  // FFI hot path: read pixels here for ML/CV.
  final yPlane = frame.getPlaneData(0);                 // Uint8List view
  final stride = frame.planeBytesPerRow(0);             // bytes per row (>= width)
  final avgLuma = frame.computeLuminance(0, 0, frame.width, frame.height);
});

getPlaneData(i) returns a direct view of native memory. Use planeBytesPerRow(i) / planePixelStride(i) to walk it correctly (rows are padded). computeLuminance is YUV/Android-only — on iOS (BGRA) it returns 0.0; read getPlaneData(0) instead.

Off-isolate worklet (heavy ML/CV)

For heavy per-frame work, run it on a background isolate so the UI never janks. The entry is a top-level/static function (Dart can't ship a closure across isolates). Pass model bytes via args — a worker isolate can't read rootBundle, so load assets on the main isolate. Register onFrame, and send results back:

// Top-level — runs on the worker isolate.
void detectorWorklet(FrameWorklet w) {
  final args = w.args as ({Uint8List model, String labels});
  final interpreter = Interpreter.fromBuffer(args.model); // tflite_flutter
  w.onFrame((frame) {
    final boxes = runModel(interpreter, frame); // heavy inference, off-main
    w.send(boxes);                              // sendable result → main
  });
}

// Main isolate:
final model =
    (await rootBundle.load('assets/model.tflite')).buffer.asUint8List();
final labels = await rootBundle.loadString('assets/labels.txt');
controller.frameResults.listen((r) => setState(() => _boxes = r as List));
await controller.setFrameWorklet(
  detectorWorklet,
  args: (model: model, labels: labels),
);

Frames cross to the worker as the same zero-copy FFI pointers (no buffer copy). Map detection boxes onto the preview with previewRectFromFrame. The example's Object Detector is a full EfficientDet-Lite0 worklet.

Keeping a frame past the callback

The Frame and its buffers are valid only during the callback. To use the data later (e.g. on another isolate), either copy it out synchronously:

final bytes = Uint8List.fromList(frame.getPlaneData(0)); // owns a copy

…or extend the native lifetime by balancing the ref-count exactly once:

frame.incrementRefCount();              // keep the buffer alive
// ... use frame asynchronously ...
frame.decrementRefCount();              // release it (mandatory)

Drawing ML detection boxes on the preview

Detection boxes are in the frame's coordinate space; the preview is rotated (and maybe mirrored) for display, and the two are tracked independently — so a box drawn naively drifts as the device turns. Map it with previewRectFromFrame:

final previewRect = controller.previewRectFromFrame(
  detection.boundingBox,                      // normalized 0..1 from your model
  sourceRotationDegrees: frame.orientation.degrees, // rotation you fed the model
);
// Scale previewRect onto the preview widget's rect (use displayPreviewSize for BoxFit).

It accounts for the live previewRotation and the front-camera mirror, so boxes stay aligned at any orientation. See the example's Object Detector page for a full EfficientDet-Lite0 overlay.

High-performance native C/C++ plugin

For the heaviest work, hook the synchronous frame loop in C++ (src/VisionCamera.hpp):

#include "VisionCamera.hpp"

class BrightnessPlugin : public vision::Plugin {
public:
  const char* name() const override { return "brightness"; }
  void onFrame(const vision::Frame& frame) override {
    const uint8_t* y = frame.data();          // Y / first plane
    int stride = frame.bytesPerRow();
    // ...heavy SIMD/AI math on the camera thread...
  }
};

// Register once (e.g. from an init function you call via FFI at startup):
extern "C" __attribute__((visibility("default"))) void registerMyPlugins() {
  vision::Registry::instance().addPlugin(std::make_shared<BrightnessPlugin>());
}

Put your .cpp alongside the plugin sources; it links via CMake on Android and the podspec on iOS. See src/VisionCamera_NativePluginExample.cpp for a full working example.

Platform Support

Capability	Android	iOS
Preview (GPU texture)	✅	✅
Photo capture	✅	✅
Video recording (+ audio)	✅ CameraX	✅ AVAssetWriter
Barcode / QR scanning	✅ MLKit	✅ Vision
Frame processor (Dart, main isolate)	✅ YUV planes	✅ BGRA
Frame worklet (Dart, background isolate)	✅	✅
Native C/C++ plugin (camera thread)	✅	✅
Zoom / torch / exposure / tap-focus	✅	✅
Manual focus distance	⬜	✅
takeSnapshot	⬜	✅
regionOfInterest (scanning)	⬜	✅
Minimum OS	minSdk 21	iOS 13.0

Legend: ✅ supported · ⬜ not implemented yet. Symbology coverage differs slightly between MLKit (Android) and Vision (iOS).

Limitations & Roadmap

• No web / desktop — Android + iOS only.
• Off-isolate worklets (setFrameWorklet) run heavy work on a background isolate; the simpler setFrameProcessor stays main-isolate by design (light work). Worklet entries must be top-level functions (a Dart isolate constraint).
• Recording options (RecordVideoOptions, codec/HDR) and pause/resume are not yet wired on all platforms; startRecording takes a path string.
• Mirror is set at initialize time (no runtime toggle yet).
• Controls (setZoom/setTorch/etc.) are no-ops until setActive(true).
• No RAW capture / multi-camera; takeSnapshot is iOS-only.
• Planned API polish: typed CameraException, TorchMode enum, switchCamera, richer error reporting.

Credits & Attribution

Inspired by react-native-vision-camera by Marc Rousavy — bringing the same high-performance, low-level camera philosophy to Flutter via synchronous FFI and Texture rendering.

License

MIT