Augen - Flutter AR Plugin

A cross-platform Flutter plugin for building AR (Augmented Reality) apps using ARCore on Android and RealityKit on iOS. Write your AR logic entirely in Dart — no native code required.

Features

Plane Detection — detect horizontal and vertical surfaces
3D Objects — place primitives (sphere, cube, cylinder) or load custom models (GLTF, GLB, OBJ, USDZ)
Hit Testing — tap-to-place objects on detected surfaces
Image Tracking — track real-world images and anchor content to them
Face Tracking — detect faces with facial landmarks and expressions
Cloud Anchors — persist and share AR anchors across sessions and devices
Occlusion — depth, person, and plane occlusion for realistic rendering
Physics — dynamic, static, and kinematic bodies with forces, impulses, and constraints
Multi-User AR — shared sessions with real-time object synchronization
Lighting & Shadows — directional, point, spot, and ambient lights with configurable shadows
Environmental Probes — realistic reflections and environmental lighting
Animations — skeletal animations with blending, transitions, and state machines

Lighting & occlusion are natively backed on both platforms. Capability queries (getLightingCapabilities, getOcclusionCapabilities) and config (setLightingConfig, setOcclusionConfig, setOcclusionEnabled) report real device capabilities — RealityKit lights and ARKit people-occlusion on iOS, the ARCore Depth API on Android — and degrade gracefully when unavailable.

Since 1.5.0, every feature is implemented natively on iOS with ARKit/RealityKit — see iOS Feature Support for how each one maps to Apple's frameworks.

For detailed API docs and advanced usage, see Documentation.md.

Platform Requirements

Platform	Minimum Version	AR Framework
Android	API 24 (Android 7.0)	ARCore
iOS	iOS 13.0	RealityKit & ARKit

SDK: Flutter >= 3.3.0, Dart >= 3.9.2

Installation

dependencies:
  augen: ^1.5.0

flutter pub get

Android Setup

Add to your android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera.ar" android:required="true" />
<uses-feature android:glEsVersion="0x00030000" android:required="true" />

<application>
    <meta-data android:name="com.google.ar.core" android:value="required" />
</application>

Set minSdkVersion to at least 24 in android/app/build.gradle.

iOS Setup

Add to your ios/Runner/Info.plist:

<key>NSCameraUsageDescription</key>
<string>This app requires camera access for AR features</string>

<key>UIRequiredDeviceCapabilities</key>
<array>
    <string>arkit</string>
</array>

Set the deployment target to at least iOS 13.0.

Multi-user AR (optional): sessions use Apple's MultipeerConnectivity over the local network. If you use the multi-user API, also add:

<key>NSLocalNetworkUsageDescription</key>
<string>Used to discover nearby devices and share AR sessions.</string>

<key>NSBonjourServices</key>
<array>
    <string>_augen-ar._tcp</string>
    <string>_augen-ar._udp</string>
</array>

Swift Package Manager: Augen ships with both Swift Package Manager and CocoaPods support, so it works whether or not your app has migrated to SPM. No extra steps are required — the Flutter tool wires up the correct integration automatically. If you have enabled SPM (flutter config --enable-swift-package-manager), augen is resolved as a Swift Package; otherwise it is installed via CocoaPods.

Quick Start

1. Display the AR View

AugenView is the widget that renders the camera feed and AR scene. When the view is ready, you receive an AugenController to drive everything.

import 'package:flutter/material.dart';
import 'package:augen/augen.dart';

class ARScreen extends StatefulWidget {
  @override
  State<ARScreen> createState() => _ARScreenState();
}

class _ARScreenState extends State<ARScreen> {
  AugenController? _controller;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: AugenView(
        onViewCreated: (controller) {
          _controller = controller;
          _initAR();
        },
        config: ARSessionConfig(
          planeDetection: true,
          lightEstimation: true,
        ),
      ),
    );
  }

  Future<void> _initAR() async {
    final supported = await _controller!.isARSupported();
    if (!supported) return;

    await _controller!.initialize(
      ARSessionConfig(planeDetection: true, lightEstimation: true),
    );

    // React to detected planes
    _controller!.planesStream.listen((planes) {
      debugPrint('Detected ${planes.length} planes');
    });
  }

  @override
  void dispose() {
    _controller?.dispose();
    super.dispose();
  }
}

2. Place Objects via Hit Test

Tap a detected surface to place a 3D object:

final results = await _controller!.hitTest(screenX, screenY);
if (results.isNotEmpty) {
  await _controller!.addNode(
    ARNode(
      id: 'sphere_1',
      type: NodeType.sphere,
      position: results.first.position,
      scale: Vector3(0.1, 0.1, 0.1),
    ),
  );
}

3. Load Custom 3D Models

// From Flutter assets
await _controller!.addModelFromAsset(
  id: 'ship',
  assetPath: 'assets/models/spaceship.glb',
  position: Vector3(0, 0, -1),
  scale: Vector3(0.1, 0.1, 0.1),
);

// From a URL
await _controller!.addModelFromUrl(
  id: 'building',
  url: 'https://example.com/models/building.glb',
  position: Vector3(1, 0, -2),
  modelFormat: ModelFormat.glb,
);

Recommended formats: GLB for Android, USDZ for iOS. OBJ works on both. On iOS, RealityKit cannot read GLTF/GLB — those nodes render as an orange placeholder box (and log a warning), so ship USDZ for iOS.

iOS Feature Support

Every Augen API is backed natively on iOS. Where ARKit/RealityKit has no direct equivalent, the behaviour is emulated and the Dart-visible state stays consistent.

Feature	iOS implementation	Notes
Plane detection, hit test	`ARWorldTrackingConfiguration`, `ARView.raycast`	Horizontal + vertical planes
Custom models	RealityKit (USDZ/Reality), ModelIO (OBJ)	GLTF/GLB → placeholder
Animations	RealityKit playback for baked clips + built-in procedural clips (`idle`, `walk`, `run`, `jump`, `spin`, `bounce`, `pulse`, `wobble`)	Blending, crossfades, blend trees and state machines work on both; bone masks are stored but not applied
Physics	RealityKit `PhysicsBodyComponent`	Joints/constraints and cross-node collisions need iOS 18+
Image tracking	`ARReferenceImage` detection (assets, files, URLs or bytes)	Targets are validated by ARKit before use
Face tracking	`ARFaceTrackingConfiguration`	TrueDepth front camera; switches the session to the front camera
Lighting & shadows	RealityKit directional/point/spot lights + ARKit light estimation	Ambient light comes from environment texturing
Occlusion	ARKit person segmentation (with depth on supported devices)
Environmental probes	ARKit environment texturing
Cloud anchors	`ARWorldMap` saved on device	No cloud backend; sharing needs a mapped scene — move the device around first
Multi-user	MultipeerConnectivity + ARKit collaboration	Nearby devices on the same local network

Web Marker-Based AR

Augen runs in the browser via Flutter Web (Wasm or JS) and detects square visual markers in the live camera feed using an image-template matching detector implemented in JavaScript with dart:js_interop bindings.

Status: The detector matches PNG/JPG marker templates with multi-scale normalized cross-correlation (NCC). Pose translation is computed from apparent marker size and the configured physical width; rotation is currently identity. For full 6-DoF pose, a future release will bundle ARToolKit Wasm or OpenCV.js POSIT. Three.js rendering integration is also planned — the renderer already maintains scene state and marker transforms, but does not yet draw geometry. Contributions welcome.

Browser Support

Browser	Support
Chrome (desktop & Android)	✅ Full
Edge	✅ Full
Safari	⚠️ Partial (WebRTC limitations)
Firefox	⚠️ Partial (WebAssembly SIMD varies)

Requirements

Flutter Wasm: flutter run -d chrome --wasm
HTTPS required for getUserMedia camera access (localhost is exempt)
Camera permission must be granted by the user
Server must serve .wasm files with Content-Type: application/wasm
For SharedArrayBuffer (optional perf boost): set Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp headers

Marker Target Types

Type	Description	Status
`ARMarkerType.pattern` (with `imagePath`)	PNG/JPG image template matched via NCC	✅ Implemented
`ARMarkerType.pattern` (with `patternPath`)	Classic ARToolKit `.patt` file	⚠️ Planned (ARToolKit Wasm)
`ARMarkerType.barcode`	Numeric barcode markers	⚠️ Planned
`ARMarkerType.aruco`	ArUco dictionary markers	⚠️ Planned

Quick Start

AugenView(
  config: const ARSessionConfig(
    markerTracking: true,
    planeDetection: false,
  ),
  onViewCreated: (controller) async {
    await controller.initialize(
      const ARSessionConfig(
        markerTracking: true,
        markerDetectionOptions: ARMarkerDetectionOptions(
          maxDetectionFps: 20,
          debug: true,
        ),
      ),
    );

    // Use a PNG/JPG image as the marker template.
    await controller.addMarkerTarget(
      const ARMarkerTarget(
        id: 'hiro',
        name: 'Hiro marker',
        type: ARMarkerType.pattern,
        imagePath: 'assets/markers/Hiro_marker.png',
        physicalWidth: 0.08, // 8 cm
      ),
    );

    await controller.setMarkerTrackingEnabled(true);

    controller.trackedMarkersStream.listen((markers) {
      for (final marker in markers) {
        if (marker.isTracked && marker.isReliable) {
          // Anchor content to the marker
        }
      }
    });
  },
);

A standalone runnable demo lives at example/web_marker_ar/:

cd example/web_marker_ar
flutter run -d chrome --wasm

Supported vs Unsupported Web Features

Feature	Web Support
Marker tracking (PNG/JPG image templates)	✅
Camera feed rendering	✅
`trackedMarkersStream` & marker pose updates	✅
3D node anchoring to markers (scene-graph state)	✅
Three.js geometry rendering	⚠️ Planned
Full 6-DoF pose (rotation)	⚠️ Planned (ARToolKit Wasm)
`.patt` / barcode / ArUco marker types	⚠️ Planned
Plane detection	❌ Mobile only
Image tracking (ARCore/ARKit style)	❌ Mobile only
Face tracking	❌ Mobile only
Cloud anchors	❌ Mobile only
Occlusion / Physics / LiDAR	❌ Mobile only

Performance Tips

Set maxDetectionFps to 15–20 to balance accuracy and CPU usage
Use small marker pattern files (< 10 KB)
Print markers at the configured physicalWidth for correct scale
Prefer Chrome/Edge for best WebAssembly performance
Use debug: false in production to disable visual overlays

Troubleshooting

Camera not starting: Ensure HTTPS (or localhost) and that the user granted camera permission.
Marker not detected: Verify the imagePath (PNG/JPG) is declared in pubspec.yaml assets and the printed marker matches physicalWidth. Ensure adequate lighting and that the marker fills a reasonable portion of the camera frame.
Camera hangs at "Initializing…": On laptops with no rear camera, facingMode: 'environment' falls back to any available camera automatically. If it still hangs, close other apps/tabs holding the webcam.
Wasm not loading: Confirm the server serves .wasm with Content-Type: application/wasm.
SharedArrayBuffer errors: Add COOP/COEP headers to your web server.
Low FPS: Reduce maxDetectionFps, close other tabs, or use a device with better GPU.

Architecture Overview

augen/
  lib/
    augen.dart                  # Public barrel export
    src/
      augen_controller.dart     # AugenController — all AR operations
      augen_view.dart           # AugenView widget
      models/                   # Data classes (ARNode, ARPlane, Vector3, etc.)
  android/                      # Kotlin — ARCore integration
  ios/                          # Swift — RealityKit / ARKit integration
  example/                      # Full-featured demo app
  test/                         # Unit and integration tests

How it works: AugenView creates a platform view (Android: PlatformViewLink, iOS: UiKitView) that hosts the native AR renderer. All communication between Dart and native happens through Flutter method channels, abstracted behind AugenController.

Key Classes

Class	Purpose
`AugenView`	Widget that displays the AR camera + scene
`AugenController`	Controls the AR session — add/remove nodes, hit test, manage anchors, animations, physics, etc.
`ARSessionConfig`	Session options: plane detection, light estimation, depth data, auto focus
`ARNode`	A 3D object in the scene (primitives or custom models)
`ARPlane`	A detected surface (horizontal/vertical)
`ARAnchor`	A fixed point in world space
`ARHitResult`	Result of a raycast against detected geometry
`Vector3` / `Quaternion`	3D position and rotation types

Reactive Streams

AugenController exposes streams for all AR state updates. Subscribe to stay in sync:

_controller.planesStream          // detected planes
_controller.anchorsStream         // anchors
_controller.trackedImagesStream   // image tracking results
_controller.facesStream           // face tracking results
_controller.errorStream           // errors
_controller.physicsBodiesStream   // physics body updates
_controller.lightsStream          // light changes
// ... and more — see Documentation.md for the full list

Example Apps

The example/ directory contains a complete demo app with tabs for every feature (planes, nodes, images, faces, cloud anchors, occlusion, physics, multi-user, lighting, probes, animations).

cd example
flutter run

Web Marker AR Example

A standalone web example demonstrating marker-based AR is available at example/web_marker_ar/:

cd example/web_marker_ar
flutter run -d chrome --wasm

See the Web Marker AR README for details on setup and usage.

Testing

# Run all tests
flutter test

# Run specific test suites
flutter test test/augen_controller_test.dart
flutter test test/augen_animation_test.dart

# Integration tests (requires a device or simulator)
cd example
flutter test integration_test/plugin_integration_test.dart

# Strict on-device test of every iOS feature (physical iPhone required)
flutter test integration_test/ios_features_e2e_test.dart -d <device-id>

Troubleshooting

Android — ARCore not working:

Verify the device supports ARCore
Ensure Google Play Services for AR is installed
Confirm minSdkVersion >= 24

iOS — ARKit not available:

Requires A9 chip or later (iPhone 6s+)
Confirm deployment target is iOS 13.0+
Ensure the arkit capability is declared in Info.plist

iOS — debug build shows "can only be launched from Flutter tooling":

Debug builds must be started by flutter run or Xcode, not from the home screen
If flutter run hangs waiting for the VM Service, close Xcode and run flutter config --enable-lldb-debugging, or use flutter run --release

iOS — multi-user finds no peers:

Add NSLocalNetworkUsageDescription and NSBonjourServices (see iOS Setup)
Allow the Local Network prompt; both devices must be on the same network

Camera permission denied (both platforms):

Add the required permission entries listed in the setup sections above
Request runtime permission before showing the AR view

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/my-feature)
Commit your changes
Push and open a Pull Request

License

MIT — see LICENSE for details.

Support

If Augen has helped your project, you can support continued development with a tip on Solana:

Solana wallet address:

ApeGNaqfRM3HJocYHfdZ88GmFj2wqVCxUzm5MGvGNrWy

SOL and SPL tokens (USDC, etc.) on Solana mainnet are welcome. Thanks for your support! 🙏

Augen - Flutter AR Plugin

Features

Platform Requirements

Installation

Android Setup

iOS Setup

Quick Start

1. Display the AR View

2. Place Objects via Hit Test

3. Load Custom 3D Models

iOS Feature Support

Web Marker-Based AR

Browser Support

Requirements

Marker Target Types

Quick Start

Supported vs Unsupported Web Features

Performance Tips

Troubleshooting

Architecture Overview

Key Classes

Reactive Streams

Example Apps

Web Marker AR Example

Testing

Troubleshooting

Contributing

License

Links

Support

Libraries

augen package