hand_detection

Platform Language: Dart
Pub Version pub points CI Tests License

Flutter implementation of Google's MediaPipe hand detection and landmark models using TensorFlow Lite. Completely local: no remote API, just pure on-device, offline detection.

Hand Detection with 21-Point Landmarks

Hand detection example

Features

  • On-device hand detection, runs fully offline
  • 21-point hand landmarks with 3D depth information (x, y, z coordinates)
  • Handedness detection (left/right hand)
  • Gesture recognition: closed fist, open palm, pointing up, thumbs down, thumbs up, victory, I love you
  • Truly cross-platform: compatible with Android, iOS, macOS, Windows, and Linux
  • The example app illustrates how to detect and render results on images

Quick Start

import 'dart:io';
import 'package:hand_detection/hand_detection.dart';

Future main() async {
  final detector = await HandDetector.create();

  final imageBytes = await File('path/to/image.jpg').readAsBytes();
  List<Hand> hands = await detector.detect(imageBytes);

  for (final hand in hands) {
    final boundingBox = hand.boundingBox;
    final handedness = hand.handedness;

    if (hand.hasLandmarks) {
      final wrist = hand.getLandmark(HandLandmarkType.wrist);
      final indexTip = hand.getLandmark(HandLandmarkType.indexFingerTip);
      print('Wrist: (${wrist?.x}, ${wrist?.y})');
    }
  }

  await detector.dispose();
}

Performance

Hardware Acceleration

The package automatically selects the best acceleration strategy for each platform:

Platform Default Delegate Speedup Notes
macOS XNNPACK 2-5x SIMD vectorization (NEON on ARM, AVX on x86)
Linux XNNPACK 2-5x SIMD vectorization
iOS Metal GPU 2-4x Hardware GPU acceleration
Android XNNPACK 2-5x ARM NEON SIMD acceleration
Windows XNNPACK 2-5x SIMD vectorization (AVX on x86)

No configuration needed, just call initialize() and you get the optimal performance for your platform.

Advanced Performance Configuration

// Auto mode (default), optimal for each platform
final detector = await HandDetector.create();

// Force XNNPACK (all native platforms)
final detector = await HandDetector.create(
  performanceConfig: PerformanceConfig.xnnpack(numThreads: 4),
);

// Force GPU delegate (iOS recommended, Android experimental)
final detector = await HandDetector.create(
  performanceConfig: PerformanceConfig.gpu(),
);

// CPU-only (maximum compatibility)
final detector = await HandDetector.create(
  performanceConfig: PerformanceConfig.disabled,
);

Advanced: Direct Mat Input

If you already have a decoded cv.Mat from another OpenCV pipeline, pass it directly:

import 'package:hand_detection/hand_detection.dart';

Future<void> processFrame(Mat frame) async {
  final detector = await HandDetector.create();

  final hands = await detector.detectFromMat(frame);

  frame.dispose(); // always dispose Mats after use
  await detector.dispose();
}

For live camera streams, prefer prepareCameraFrame + detectFromCameraFrame (see below): it keeps cvtColor / rotate / downscale off the UI thread.

Bounding Boxes

The boundingBox property returns a BoundingBox object representing the hand bounding box in absolute pixel coordinates. The BoundingBox provides convenient access to corner points, dimensions (width and height), and the center point.

Accessing Corners

final BoundingBox boundingBox = hand.boundingBox;

// Access individual corners by name (each is a Point with x and y)
final Point topLeft     = boundingBox.topLeft;       // Top-left corner
final Point topRight    = boundingBox.topRight;      // Top-right corner
final Point bottomRight = boundingBox.bottomRight;   // Bottom-right corner
final Point bottomLeft  = boundingBox.bottomLeft;    // Bottom-left corner

// Access coordinates
print('Top-left: (${topLeft.x}, ${topLeft.y})');

Additional Bounding Box Parameters

final BoundingBox boundingBox = hand.boundingBox;

// Access dimensions and center
final double width  = boundingBox.width;     // Width in pixels
final double height = boundingBox.height;    // Height in pixels
final Point center = boundingBox.center;  // Center point

// Access coordinates
print('Size: ${width} x ${height}');
print('Center: (${center.x}, ${center.y})');

// Access all corners as a list (order: top-left, top-right, bottom-right, bottom-left)
final List<Point> allCorners = boundingBox.corners;

Hand Landmarks (21-Point)

The landmarks property returns a list of 21 HandLandmark objects representing key points on the detected hand. Each landmark has 3D coordinates (x, y, z) and a visibility score.

21 Hand Landmarks

Index Landmark Description
0 wrist Wrist
1-4 thumbCMC, thumbMCP, thumbIP, thumbTip Thumb joints and tip
5-8 indexFingerMCP, indexFingerPIP, indexFingerDIP, indexFingerTip Index finger
9-12 middleFingerMCP, middleFingerPIP, middleFingerDIP, middleFingerTip Middle finger
13-16 ringFingerMCP, ringFingerPIP, ringFingerDIP, ringFingerTip Ring finger
17-20 pinkyMCP, pinkyPIP, pinkyDIP, pinkyTip Pinky finger

Accessing Landmarks

final Hand hand = hands.first;

// Access specific landmarks by type
final wrist = hand.getLandmark(HandLandmarkType.wrist);
final indexTip = hand.getLandmark(HandLandmarkType.indexFingerTip);
final thumbTip = hand.getLandmark(HandLandmarkType.thumbTip);

if (wrist != null) {
  print('Wrist: (${wrist.x}, ${wrist.y}, ${wrist.z})');
  print('Visibility: ${wrist.visibility}');
}

// Iterate through all landmarks
for (final landmark in hand.landmarks) {
  print('${landmark.type.name}: (${landmark.x}, ${landmark.y})');
}

Drawing Hand Skeleton

Use the handLandmarkConnections constant to draw the hand skeleton:

import 'package:hand_detection/hand_detection.dart';

// Draw skeleton connections
for (final connection in handLandmarkConnections) {
  final start = hand.getLandmark(connection[0]);
  final end = hand.getLandmark(connection[1]);

  if (start != null && end != null) {
    canvas.drawLine(
      Offset(start.x, start.y),
      Offset(end.x, end.y),
      paint,
    );
  }
}

Handedness

The handedness property indicates whether the detected hand is a left or right hand:

final Hand hand = hands.first;

if (hand.handedness == Handedness.left) {
  print('Left hand detected');
} else if (hand.handedness == Handedness.right) {
  print('Right hand detected');
}

Gesture Recognition

Enable gesture recognition to classify hand poses into 7 gestures:

Gesture detection example

Gesture Description
closedFist Closed fist
openPalm Open palm
pointingUp Index finger pointing up
thumbDown Thumbs down
thumbUp Thumbs up
victory Victory / peace sign
iLoveYou "I love you" sign

Enabling Gestures

final detector = HandDetector(
  enableGestures: true,
  gestureMinConfidence: 0.5, // optional, default 0.5
);
await detector.initialize();

final hands = await detector.detect(imageBytes);
for (final hand in hands) {
  if (hand.hasGesture) {
    print('Gesture: ${hand.gesture!.type.name}');
    print('Confidence: ${hand.gesture!.confidence}');
  }
}

Gesture recognition uses a two-stage pipeline (gesture embedder + classifier) and requires HandMode.boxesAndLandmarks (the default mode).

Detection Modes

This package supports two detection modes:

Mode Features Speed
boxesAndLandmarks (default) Bounding boxes + 21 landmarks + handedness Standard
boxes Bounding boxes only Faster

Code Examples

// Full mode (default): bounding boxes + 21 landmarks + handedness
final detector = HandDetector(
  mode: HandMode.boxesAndLandmarks,
);

// Fast mode: bounding boxes only
final detector = HandDetector(
  mode: HandMode.boxes,
);

Configuration Options

The HandDetector constructor accepts several configuration options:

final detector = HandDetector(
  mode: HandMode.boxesAndLandmarks,      // Detection mode
  landmarkModel: HandLandmarkModel.full, // Landmark model variant
  detectorConf: 0.45,                     // Palm detection confidence (0.0-1.0)
  maxDetections: 10,                     // Maximum hands to detect
  minLandmarkScore: 0.5,                 // Minimum landmark confidence (0.0-1.0)
  interpreterPoolSize: 1,                // TFLite interpreter pool size
  performanceConfig: const PerformanceConfig(),    // Performance config (default: auto)
  enableGestures: false,                 // Enable gesture recognition
  gestureMinConfidence: 0.5,             // Minimum gesture confidence (0.0-1.0)
);

Live Camera Detection

For real-time hand detection from a camera feed, use detectFromCameraImage. All processing runs off the UI thread.

Desktop (Windows / macOS / Linux): The default camera package does not include a streaming implementation for desktop platforms. You must also add camera_desktop to your pubspec.yaml, otherwise startImageStream throws UnimplementedError: onStreamedFrameAvailable() is not implemented.

dependencies:
  camera: ^0.12.0
  camera_desktop: ^1.1.6   # required for Windows, macOS, and Linux streaming

import 'package:camera/camera.dart';
import 'package:hand_detection/hand_detection.dart';

final detector = await HandDetector.create();

final cameras = await availableCameras();
final camera = CameraController(
  cameras.first,
  ResolutionPreset.medium,
  enableAudio: false,
  imageFormatGroup: ImageFormatGroup.yuv420, // prevents JPEG fallback on Android; ignored on desktop
);
await camera.initialize();

camera.startImageStream((CameraImage image) async {
  final hands = await detector.detectFromCameraImage(
    image,
    // rotation: rotationForFrame(...), // recommended on Android/iOS
    maxDim: 640,
  );
  // Process hands...
});

Tips:

  • Pass rotation: on Android/iOS so the detector sees upright frames. Use rotationForFrame(...) to compute the correct value from sensor orientation and device orientation. On desktop frames are always upright so omit it.
  • Pass maxDim: 640 to downscale frames before inference. Recommended: full-res frames waste bandwidth since the model input is much smaller.
  • Mirror the overlay on the front camera to match CameraPreview's auto-mirrored texture.
  • For advanced use, prepareCameraFrame(...) + detectFromCameraFrame(...) is the lower-level two-step API.

See the full example app for a complete implementation.

Background Processing

All inference runs automatically in a background isolate: the UI thread is never blocked during detection or gesture recognition. No special configuration is needed; HandDetector handles isolate management internally.

Example

The sample code from the pub.dev example tab includes a Flutter app that paints detections onto an image: bounding boxes and 21-point hand landmarks with skeleton connections.

Libraries

hand_detection
On-device hand detection and landmark estimation using TensorFlow Lite.