air_pointer

Platform-agnostic canvas input abstraction for Flutter. Ships a MouseInputSource for mouse/trackpad/touch and a GestureInputSource powered by MediaPipe HandLandmarker for touchless hand control (Flutter Web only), both unified behind a single CanvasInputController — the canvas never knows which source the events came from.

Quick start

import 'package:air_pointer/air_pointer.dart';

class MyCanvasState extends State<MyCanvas> {
  late final CanvasInputController _controller;
  late final StreamSubscription<PointerInputEvent> _sub;

  @override
  void initState() {
    super.initState();
    _controller = CanvasInputController(
      sources: [
        MouseInputSource(),     // mouse, trackpad, touch
        GestureInputSource(),   // MediaPipe hand tracking (web only; no-op elsewhere)
      ],
    );
    _sub = _controller.events.listen(_onInput);
  }

  void _onInput(PointerInputEvent event) {
    switch (event) {
      case CanvasTapEvent(:final position):
        // resolved tap (no drag)
      case CanvasDoubleTapEvent(:final position):
        // second tap within the double-tap window (~300 ms)
        // always preceded by CanvasTapEvent on the same frame
      case CanvasLongPressEvent(:final position):
        // pointer held still beyond the long-press threshold
      case CanvasDownEvent(:final position):
        // drag/pinch started
      case CanvasMoveEvent(:final position):
        // drag in progress
      case CanvasUpEvent(:final position):
        // released — commit the action
      case CanvasCancelEvent():
        // drag interrupted (e.g. hand left the camera mid-drag) — discard it
      case CanvasHoverEvent(:final position):
        // cursor hovering (no button held)
      case CanvasScrollEvent(:final position, :final delta, :final isTrackpad):
        // scroll wheel or pointing-finger scroll
      case CanvasScaleEvent(:final focalPoint, :final scaleDelta, :final panDelta):
        // pinch-to-zoom (two fingers or two hands)
      case CanvasScaleEndEvent():
        // scale gesture ended
      case CanvasSwipeEvent(:final direction, :final velocity):
        // fast directional motion in an open hand
      case CanvasGestureEvent(:final gesture, :final isSecondHand):
        // discrete gesture recognised (thumbsUp, victory, etc.)
        // isSecondHand is true when it came from the second detected hand
    }
  }

  @override
  Widget build(BuildContext context) => _controller.buildSurface(
        child: MyCanvasPainter(...),
      );

  @override
  void dispose() {
    unawaited(_sub.cancel());
    unawaited(_controller.dispose());
    super.dispose();
  }
}

Event type reference

Type Fields Description
CanvasTapEvent position Resolved tap (no drag)
CanvasDoubleTapEvent position Second tap within the double-tap window; always preceded by CanvasTapEvent
CanvasLongPressEvent position Pointer held still beyond the long-press threshold
CanvasDownEvent position Drag/pinch started
CanvasMoveEvent position Drag in progress
CanvasUpEvent position Drag ended — commit
CanvasCancelEvent — Drag aborted — discard
CanvasHoverEvent position Hover (no press)
CanvasScrollEvent position, delta, isTrackpad Scroll wheel or pointing-finger scroll
CanvasScaleEvent focalPoint, scaleDelta, panDelta, rotation Pinch/spread with optional rotation
CanvasScaleEndEvent — Scale gesture ended
CanvasSwipeEvent direction, velocity Fast directional cursor movement
CanvasGestureEvent gesture, isSecondHand Discrete hand gesture (edge-triggered on change)

GestureInputSource (Flutter Web only)

1. Add the MediaPipe CDN script

In web/index.html, before flutter_bootstrap.js:

<script src="https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.21/vision_bundle.js"
        crossorigin="anonymous"></script>
<script src="flutter_bootstrap.js" async></script>

2. Copy the worker script

Place hand_tracker_worker.js (from example/web/) next to your index.html. The worker runs MediaPipe inference off the main thread.

3. Initialize the source

final _gestureSource = GestureInputSource(
  onError: (e, st) => print('hand tracking: $e'),
  dwellDuration: const Duration(milliseconds: 700),  // enable dwell-tap
  doubleTapWindow: const Duration(milliseconds: 300),
  longPressDuration: const Duration(milliseconds: 1200),
  scrollEnabled: true,
  maxHands: 2,  // set to 1 for better performance in single-hand apps
);

@override
void didChangeDependencies() {
  super.didChangeDependencies();
  // Pass the canvas size so cursor coordinates are in screen pixels.
  _gestureSource.updateCanvasSize(MediaQuery.sizeOf(context));
  unawaited(_gestureSource.initialize());
}

initialize() is idempotent — safe to call from didChangeDependencies.

Gesture mapping

Hand gesture Event
Open hand, fingertip moves CanvasHoverEvent
Pinch (thumb + index < 5 % gap) CanvasDownEvent
Hold pinch + move CanvasMoveEvent
Release pinch CanvasUpEvent
Hand exits frame mid-drag CanvasCancelEvent (discard, not commit)
Cursor holds still for dwellDuration CanvasTapEvent
Second dwell within doubleTapWindow CanvasTapEvent + CanvasDoubleTapEvent
Cursor holds still for longPressDuration CanvasLongPressEvent
Index finger extended, move up/down CanvasScrollEvent (when scrollEnabled: true)
Fast directional motion, open hand CanvasSwipeEvent (when swipeThreshold > 0)
Two-hand spread/pinch CanvasScaleEvent with rotation
Two hands separate CanvasScaleEndEvent
Recognised discrete gesture CanvasGestureEvent (edge-triggered; isSecondHand for the second hand)

Position is smoothed with a OneEuroFilter and the x-axis is mirrored so motion feels natural facing the front camera.

Dwell-tap

Dwell-tap lets users click without a physical button — the cursor fires CanvasTapEvent after holding still within dwellRadius pixels for dwellDuration:

GestureInputSource(
  dwellDuration: const Duration(milliseconds: 700),
  dwellRadius: 12.0,
)

GestureDebugInfo.dwellProgress (0–1) drives a progress ring in the example app. Progress is preserved through the grace window so brief occlusions don't reset it.

Double-tap and long-press

Both GestureInputSource and MouseInputSource fire the same events:

GestureInputSource(
  dwellDuration: const Duration(milliseconds: 500),
  doubleTapWindow: const Duration(milliseconds: 300),  // time between two taps
  longPressDuration: const Duration(milliseconds: 1200),
)

MouseInputSource()  // double-tap via DateTime gap; long-press via GestureDetector (~500 ms)

CanvasDoubleTapEvent is always preceded by CanvasTapEvent on the same frame so consumers that only handle single tap continue to work.

Discrete gesture events

CanvasGestureEvent fires once when a RecognizedGesture value first appears, then rearms when the hand is lost so re-entry with the same gesture fires again:

case CanvasGestureEvent(:final gesture, :final isSecondHand):
  if (!isSecondHand) {
    switch (gesture) {
      case RecognizedGesture.thumbUp:   _undo();
      case RecognizedGesture.victory:   _redo();
      case RecognizedGesture.openPalm:  _showMenu();
      default: break;
    }
  }

Supported values: thumbUp, thumbDown, openPalm, closedFist, victory, pointingUp, iLoveYou, none.

Camera preview

_gestureSource.buildCameraPreview(width: 240, height: 180)

Returns a widget with a live mirrored feed. Shows a placeholder until the camera is ready, and an error card if initialization failed.

Debug overlay

Subscribe to GestureInputSource.debugInfo for per-frame GestureDebugInfo snapshots (phase, pinch distance, dwell progress, landmarks, detected gestures, worker latency). Use them to build a custom debug overlay — the example app (example/) has a ready-made one.

MouseInputSource options

MouseInputSource(
  tapSlop: 10.0,           // max displacement (px) still treated as a tap
  scrollMultiplier: 1.0,   // scale scroll deltas (e.g. 1/zoomLevel)
)

CanvasScrollEvent.isTrackpad is true for macOS/iOS two-finger pan; the OS already applies momentum so consumers should skip extra inertia for those events.

Double-tap is detected via a 300 ms gap between tap releases. Long-press uses Flutter's GestureDetector.onLongPressStart (~500 ms threshold). If a drag was in progress when long-press fires, CanvasCancelEvent is emitted first.

TouchInputSource (Android / iOS)

MouseInputSource works on mobile but maps single-finger drag to element-drag events (CanvasDownEvent/CanvasMoveEvent/CanvasUpEvent). TouchInputSource uses direct-manipulation semantics: finger drags pan the canvas, pinch zooms it.

Use defaultPointerSource() to pick the right source automatically (see below), or wire it manually:

import 'package:flutter/foundation.dart' show defaultTargetPlatform, TargetPlatform;

_controller = CanvasInputController(
  sources: [
    if (defaultTargetPlatform == TargetPlatform.android ||
        defaultTargetPlatform == TargetPlatform.iOS)
      TouchInputSource()      // mobile: pan + pinch
    else
      MouseInputSource(),     // desktop / web: drag + scroll
    GestureInputSource(),     // hand tracking on web; no-op elsewhere
  ],
);

Gesture mapping

Touch gesture Event
Tap CanvasTapEvent
Double-tap CanvasTapEvent + CanvasDoubleTapEvent
Long press CanvasLongPressEvent
Single-finger drag CanvasScrollEvent (pan)
Flick / fling CanvasScrollEvent with velocity (seed momentum)
Two-finger pinch or spread CanvasScaleEvent
Two-finger release CanvasScaleEndEvent

Delta and velocity conventions

CanvasScrollEvent.delta follows the same sign convention as mouse scroll: offset -= delta pans the canvas in the direction the finger moved.

On fling, delta is Offset.zero and velocity (pixels/second) is non-zero in the direction the content should continue moving. Use it to seed a ticker:

case CanvasScrollEvent(:final delta, :final velocity, :final isTrackpad):
  if (velocity != Offset.zero) {
    _momentum = velocity;
    _momentumTicker.start();
  } else {
    _offset -= delta;
    setState(() {});
  }

Options

TouchInputSource(
  tapSlop: 10.0,          // max px displacement still treated as a tap
  doubleTapWindow: const Duration(milliseconds: 300),
)

StylusInputSource (Apple Pencil / S-Pen)

StylusInputSource handles PointerDeviceKind.stylus and PointerDeviceKind.invertedStylus (eraser end). It uses the same element-drag semantics as MouseInputSource and filters out mouse/touch/trackpad events so it can be combined with other sources without emitting duplicate events.

final _stylus = StylusInputSource();

_controller = CanvasInputController(
  sources: [
    _stylus,            // Apple Pencil / S-Pen: drag, tap, hover
    TouchInputSource(), // finger touch: pan + pinch
  ],
);

// Distinguish pen tip from eraser
_stylus.eraserModeStream.listen((isEraser) {
  setState(() => _isEraserMode = isEraser);
});

Gesture mapping

Pen contact Event
Tap (minimal movement) CanvasTapEvent
Double-tap CanvasTapEvent + CanvasDoubleTapEvent
Pen drag CanvasDownEvent → CanvasMoveEvent → CanvasUpEvent
Hover (proximity) CanvasHoverEvent
OS pointer cancel CanvasCancelEvent

Pressure

CanvasDownEvent.pressure and CanvasMoveEvent.pressure carry the hardware value (0.0–1.0). All other input sources emit the default 1.0.

case CanvasMoveEvent(:final position, :final pressure):
  _brush.draw(position, size: pressure * _maxBrushSize);

Eraser detection

StylusInputSource.eraserModeStream emits true when the eraser end contacts the screen and false when the pen tip is active. It only emits on mode changes, not every frame.

Options

StylusInputSource(
  tapSlop: 8.0,          // max px displacement still treated as a tap (default)
  doubleTapWindow: const Duration(milliseconds: 300),
)

defaultPointerSource()

Returns the best-fit CanvasInputSource for the current platform without any manual platform checks:

_controller = CanvasInputController(
  sources: [defaultPointerSource()],
);
Platform Source returned
Android, iOS TouchInputSource
Web MouseInputSource
macOS, Windows, Linux MouseInputSource

Uses kIsWeb and defaultTargetPlatform from package:flutter/foundation.dart (not dart:io), so it is safe to call on web.

GestureInputSource is intentionally not included — add it separately when you need air-gesture input:

_controller = CanvasInputController(
  sources: [
    defaultPointerSource(),
    GestureInputSource(),   // web: MediaPipe; native: no-op stub
  ],
);

Muting a source when another is active

CanvasInputController can suppress a source whenever a second source is actively delivering events — for example, mute the mouse when the hand tracker has a confirmed hand in frame:

final _active = StreamController<bool>.broadcast();

_controller = CanvasInputController(
  sources: [MouseInputSource(), _gestureSource],
  muteWhenActive: {MouseInputSource()},  // suppress this source …
  activeStream: _active.stream,          // … while this stream emits true
);

// Drive it from the statusStream:
_gestureSource.statusStream.listen((status) {
  _active.add(status is HandTrackingTracking);
});

Both muteWhenActive and activeStream must be provided together or not at all.

Tracking lifecycle (statusStream)

GestureInputSource.statusStream emits typed HandTrackingStatus states:

initializing → cameraReady → tracking ⇄ lost
                                       ↘ error (terminal)

_gestureSource.statusStream.listen((status) {
  switch (status) {
    case HandTrackingInitializing():  // camera + model loading
    case HandTrackingCameraReady():   // camera live; scanning for hands
    case HandTrackingTracking():      // hand confirmed in frame
    case HandTrackingLost():          // hand left the frame
    case HandTrackingError(:final error):  // unrecoverable — check onError
  }
});

HandTrackingError is terminal; no further tracking/lost states are emitted after it. Listen to onError for the underlying StateError message.

Runtime filter tuning

Adjust cursor smoothing at any time without reconstructing the source:

_gestureSource.setFilterParams(
  minCutoff: 0.5,   // lower = smoother, more lag (default 1.0)
  beta: 0.1,        // higher = less lag during fast motion (default 0.05)
  predictionHorizon: const Duration(milliseconds: 16),  // look-ahead to offset filter lag
);

Call between tracking sessions for smoothest results — a mid-session call resets the filter state and may cause a brief cursor jump on the next frame.

Per-user calibration

Default thresholds (pinch close = 0.05, open = 0.08) work for most hands in good lighting. For users with small hands, unusual skin tone, or dim environments, run a quick calibration:

// 1. Collect samples via GestureCalibrator (reads pinch distance from debugInfo).
final calibrator = GestureCalibrator();

// Call addOpenSample / addCloseSample each frame from your debugInfo subscription:
// calibrator.addOpenSample(info.pinchDistance);   // while hand is open
// calibrator.addCloseSample(info.pinchDistance);  // while hand is pinching

// 2. Compute thresholds when both poses are collected.
final result = calibrator.compute(); // null if insufficient data
if (result != null) {
  gestureSource.applyCalibration(result);
}

The example app ships a CalibrationDialog widget that handles the full guided flow.

Native hand tracking (non-web)

On iOS, Android, macOS, Windows, and Linux the web worker is unavailable. Use the LandmarkProvider interface to connect any native ML backend:

class MyTFLiteProvider implements LandmarkProvider {
  final _ctrl = StreamController<HandDetectionFrame>.broadcast();

  @override
  Stream<HandDetectionFrame> get frames => _ctrl.stream;

  void onInferenceResult(List<HandLandmarkPoint> lms) {
    _ctrl.add(HandDetectionFrame(landmarks: lms));
  }

  @override
  Widget buildPreview({double? width, double? height}) => CameraPreview(...);

  @override
  void dispose() { _ctrl.close(); }
}

final source = GestureInputSource(
  landmarkProvider: MyTFLiteProvider(),
  maxHands: 1,  // query source.maxHands to configure your provider
  dwellDuration: const Duration(milliseconds: 700),
);

HandGestureRecognizer is the pure-Dart state machine that drives all event logic — it is fully unit-testable without a camera or ML model.

Self-hosting MediaPipe assets (Flutter Web only)

By default GestureInputSource loads the WASM runtime and hand model from CDN. For offline or CSP-restricted deployments, self-host both assets:

GestureInputSource(
  mediaPipeBaseUrl: '/mediapipe/',   // serves vision_bundle.js + WASM files
  modelAssetUrl: '/mediapipe/hand_landmarker.task',
)

Use scripts/download_mediapipe.sh to fetch the pinned versions into example/web/mediapipe/. Both parameters are Flutter Web only — they have no effect on native builds and do not exist on GestureInputSource native stub.

Architecture boundary

The strict invariant: no NormalizedLandmark, HandLandmarker, JSObject, or any dart:js_interop type may appear outside lib/src/gesture/. The PointerInputEvent sealed hierarchy is the only currency that crosses the boundary — all sources speak the same type.

Known limitations

  • Flutter Web only for MediaPipe. GestureInputSource requires a LandmarkProvider on native platforms. The web implementation is self-contained (MediaPipe via web worker); native requires you to wire up your own ML backend.

  • Requires a secure context. getUserMedia only works on https:// or localhost. Serving over plain http:// will produce a camera permission error.

  • Lighting sensitivity. Tracking degrades in dim or strongly backlit conditions. MediaPipe's model is generally robust across skin tones but performance may vary. Run calibration if default thresholds are unreliable.

  • First-run CDN latency without self-hosting. The WASM runtime (~4 MB) and .task model file load from CDN on first page load. Self-host by passing mediaPipeBaseUrl and modelAssetUrl to GestureInputSource (web only — see the self-hosting section below) or using scripts/download_mediapipe.sh.

  • First-run CDN latency. MediaPipe WASM (~4 MB) loads before the first frame is processed. On a cold cache this takes 2–5 seconds. Use self-hosting (see above) to eliminate the CDN dependency and reduce cold-start time.

  • CanvasCancelEvent has no position. When a hand exits the frame mid-drag, the last known position is not re-emitted. Consumers that need a "cancel at position" snapshot should cache _lastPosition from the preceding CanvasMoveEvent.

Libraries

air_pointer