mediapipeline_flutter #

A Flutter plugin for building MediaPipe-powered vision pipelines with native Android performance.

mediapipeline_flutter allows Flutter developers to use Android native MediaPipe Tasks from Flutter using a clean MethodChannel API.

The first version supports real-time hand landmark detection and basic hand gesture recognition.

✨ Features #

✅ Android native MediaPipe Tasks integration
✅ Real-time hand landmark detection
✅ Flutter CameraImage YUV420 support
✅ Built-in hand_landmarker.task model support
✅ Basic hand gesture recognition
✅ MethodChannel based native bridge
✅ ANR-safe processing pattern
✅ Simple Flutter API
✅ Easy to extend for Pose, Face, Object Detection and more

🚀 Supported Gestures #

Gesture	Description
Open Palm	Most fingers are open
Fist	All fingers are folded
One Finger	Only index finger is open
Two Fingers	Index and middle fingers are open
Thumb Up	Thumb is open and other fingers are folded
Unknown Gesture	Gesture could not be classified

📦 Installation #

Add this package to your pubspec.yaml:

dependencies:
  mediapipeline_flutter: ^0.0.1
  camera: 0.11.2+1

Then run:

flutter pub get

🤖 Android Setup #

Minimum SDK #

MediaPipe Tasks requires Android minSdk 24 or higher.

For Kotlin Gradle:

// android/app/build.gradle.kts

defaultConfig {
    minSdk = 24
}

For Groovy Gradle:

// android/app/build.gradle

defaultConfig {
    minSdkVersion 24
}

📷 Camera Permission #

Add camera permission in:

android/app/src/main/AndroidManifest.xml

<uses-permission android:name="android.permission.CAMERA" />

Example:

<manifest xmlns:android="http://schemas.android.com/apk/res/android">

    <uses-permission android:name="android.permission.CAMERA" />

    <application
        android:label="your_app"
        android:name="${applicationName}"
        android:icon="@mipmap/ic_launcher">

        <!-- MainActivity here -->

    </application>
</manifest>

🧠 Built-in Model #

This plugin includes the MediaPipe Hand Landmarker model inside the Android plugin assets.

Model location inside the plugin:

mediapipeline_flutter/android/src/main/assets/models/hand_landmarker.task

Default native model path:

models/hand_landmarker.task

So the app does not need to add the model manually.

⚡ Quick Start #

1. Import packages #

import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:mediapipeline_flutter/mediapipeline_flutter.dart';

2. Get available cameras and create camera controller #

Future<CameraController> createCameraController() async {
  final List<CameraDescription> cameras = await availableCameras();

  final CameraDescription selectedCamera = cameras.firstWhere(
    (CameraDescription camera) =>
        camera.lensDirection == CameraLensDirection.front,
    orElse: () => cameras.first,
  );

  final CameraController controller = CameraController(
    selectedCamera,
    ResolutionPreset.low,
    enableAudio: false,
    imageFormatGroup: ImageFormatGroup.yuv420,
  );

  await controller.initialize();

  return controller;
}

3. Initialize Hand Landmarker #

Future<void> initializeMediaPipeline() async {
  await MediapipelineFlutter.initializeHandLandmarker();
}

This uses the built-in native model by default.

Default values:

modelAssetPath = models/hand_landmarker.task
useFlutterAsset = false

You can also configure it manually:

Future<void> initializeMediaPipelineWithOptions() async {
  await MediapipelineFlutter.initializeHandLandmarker(
    modelAssetPath: 'models/hand_landmarker.task',
    useFlutterAsset: false,
    maxHands: 1,
    minHandDetectionConfidence: 0.5,
    minHandPresenceConfidence: 0.5,
    minTrackingConfidence: 0.5,
  );
}

4. Detect hand from camera stream #

Future<void> startHandDetectionStream({
  required CameraController controller,
  required CameraDescription selectedCamera,
}) async {
  bool isProcessing = false;
  DateTime lastProcessedAt = DateTime.fromMillisecondsSinceEpoch(0);

  await controller.startImageStream((CameraImage image) async {
    if (isProcessing) return;

    final DateTime now = DateTime.now();

    // Process one frame every 180ms to avoid UI lag and ANR.
    if (now.difference(lastProcessedAt).inMilliseconds < 180) {
      return;
    }

    lastProcessedAt = now;
    isProcessing = true;

    try {
      final HandLandmarkerResult result =
          await MediapipelineFlutter.detectCameraImage(
        image: image,
        rotationDegrees: selectedCamera.sensorOrientation,
      );

      if (result.hasHand) {
        final DetectedHand hand = result.hands.first;

        debugPrint('Gesture: ${hand.gesture}');
        debugPrint('Hand: ${hand.handedness}');
        debugPrint('Score: ${hand.score}');
        debugPrint('Landmarks: ${hand.landmarks.length}');
      }
    } catch (e) {
      debugPrint('Detection error: $e');
    } finally {
      isProcessing = false;
    }
  });
}

📌 Full Usage Example #

Future<void> detectFromCameraImage({
  required CameraImage cameraImage,
  required int rotationDegrees,
}) async {
  final HandLandmarkerResult result =
      await MediapipelineFlutter.detectCameraImage(
    image: cameraImage,
    rotationDegrees: rotationDegrees,
  );

  if (result.hasHand) {
    final DetectedHand hand = result.hands.first;

    debugPrint(hand.gesture);
    debugPrint(hand.handedness);
    debugPrint(hand.score.toString());
    debugPrint(hand.landmarks.length.toString());
  }
}

📍 Hand Landmark Data #

Each detected hand returns 21 landmarks.

class ExampleHandLandmark {
  const ExampleHandLandmark({
    required this.x,
    required this.y,
    required this.z,
  });

  final double x;
  final double y;
  final double z;
}

Example:

void printIndexFingerTip(DetectedHand hand) {
  final HandLandmark indexFingerTip = hand.landmarks[8];

  debugPrint(indexFingerTip.x.toString());
  debugPrint(indexFingerTip.y.toString());
  debugPrint(indexFingerTip.z.toString());
}

Common landmark indexes:

Index	Landmark
0	Wrist
4	Thumb Tip
8	Index Finger Tip
12	Middle Finger Tip
16	Ring Finger Tip
20	Pinky Finger Tip

🧩 API Reference #

Initialize Hand Landmarker #

Future<void> initializeHandLandmarkerExample() async {
  await MediapipelineFlutter.initializeHandLandmarker();
}

With options:

Future<void> initializeHandLandmarkerWithOptionsExample() async {
  await MediapipelineFlutter.initializeHandLandmarker(
    modelAssetPath: 'models/hand_landmarker.task',
    useFlutterAsset: false,
    maxHands: 1,
    minHandDetectionConfidence: 0.5,
    minHandPresenceConfidence: 0.5,
    minTrackingConfidence: 0.5,
  );
}

Detect from CameraImage #

Future<HandLandmarkerResult> detectFromCameraImageExample({
  required CameraImage cameraImage,
  required int rotationDegrees,
}) async {
  return MediapipelineFlutter.detectCameraImage(
    image: cameraImage,
    rotationDegrees: rotationDegrees,
  );
}

Detect from YUV420 Planes #

Future<HandLandmarkerResult> detectFromYuv420Example({
  required int width,
  required int height,
  required Uint8List yBytes,
  required Uint8List uBytes,
  required Uint8List vBytes,
  required int yRowStride,
  required int uvRowStride,
  required int uvPixelStride,
  required int rotationDegrees,
}) async {
  return MediapipelineFlutter.detectYuv420(
    width: width,
    height: height,
    y: yBytes,
    u: uBytes,
    v: vBytes,
    yRowStride: yRowStride,
    uvRowStride: uvRowStride,
    uvPixelStride: uvPixelStride,
    rotationDegrees: rotationDegrees,
  );
}

Required import for Uint8List:

import 'dart:typed_data';

Close Hand Landmarker #

Future<void> closeHandLandmarkerExample() async {
  await MediapipelineFlutter.closeHandLandmarker();
}

🛡️ Performance Tips #

For real-time camera detection, follow these rules:

Do not process every camera frame
Use frame throttling
Use a processing lock
Use low camera resolution
Do not call detection inside build()
Stop image stream before disposing camera controller
Close the landmarker when the screen is disposed

Recommended frame throttle:

bool shouldSkipFrame(DateTime lastProcessedAt) {
  final DateTime now = DateTime.now();

  return now.difference(lastProcessedAt).inMilliseconds < 180;
}

Recommended processing lock:

Future<void> processSafely({
  required Future<void> Function() task,
}) async {
  bool isProcessing = false;

  if (isProcessing) return;

  isProcessing = true;

  try {
    await task();
  } finally {
    isProcessing = false;
  }
}

🧪 Example App #

Run the example project:

cd example
flutter clean
flutter pub get
flutter run

📁 Plugin Structure #

mediapipeline_flutter/
├── android/
│   └── src/
│       └── main/
│           ├── assets/
│           │   └── models/
│           │       └── hand_landmarker.task
│           └── kotlin/
│               └── com/example/mediapipeline_flutter/
│                   └── MediapipelineFlutterPlugin.kt
├── lib/
│   └── mediapipeline_flutter.dart
└── example/
    └── lib/
        └── main.dart

🧱 Platform Support #

Platform	Status
Android	✅ Supported
iOS	🚧 Coming soon
Web	❌ Not supported
Windows	❌ Not supported
macOS	❌ Not supported
Linux	❌ Not supported

🗺️ Roadmap #

✅ Android Hand Landmarker
✅ Basic gesture recognition
✅ CameraImage YUV420 support
✅ Built-in native model asset
❌ Pose Landmarker
❌ Face Landmarker
❌ Object Detector
❌ Image Segmenter
❌ Native CameraX real-time pipeline
❌ EventChannel result stream
❌ iOS support

❓ Troubleshooting #

Model file not found #

Error example:

Unable to open file at flutter_assets/assets/models/hand_landmarker.task

Fix:

Future<void> fixModelPathExample() async {
  await MediapipelineFlutter.initializeHandLandmarker(
    modelAssetPath: 'models/hand_landmarker.task',
    useFlutterAsset: false,
  );
}

Make sure the model exists inside the plugin:

android/src/main/assets/models/hand_landmarker.task

App crashes on start #

Make sure Android minSdk is 24 or higher:

defaultConfig {
    minSdk = 24
}

Camera is not opening #

Make sure camera permission exists:

<uses-permission android:name="android.permission.CAMERA" />

Also test on a real Android device. Some emulators may not support camera image stream properly.

🤝 Contributing #

Contributions are welcome.

You can help by adding:

Pose Landmarker
Face Landmarker
Object Detector
Image Segmenter
iOS support
Better gesture classifier
Native CameraX pipeline

📄 License #

See the LICENSE file for details.

👨‍💻 Author #

Created by Nafim Ahmed

Flutter developer focused on AI vision, automation, ERP integrations, and real-time mobile applications.

mediapipeline_flutter 0.0.1 mediapipeline_flutter: ^0.0.1 copied to clipboard

Metadata