animal_detection 4.1.0

On-device animal detection, species classification, and body pose estimation using LiteRT (formerly TensorFlow Lite).

animal_detection

On-device animal detection, species classification, and body pose estimation using TensorFlow Lite. Detects animals, classifies species/breed, and extracts 24 SuperAnimal body keypoints using a multi-stage pipeline (SSD detection, MobileNetV3 classification, RTMPose/HRNet pose estimation). Completely local: no remote API, just pure on-device, offline detection.

Features

• Animal body detection with bounding box
• Species and breed classification (dog, cat, fox, bear, etc.)
• 24-point SuperAnimal body pose estimation (spine, neck, tail, limbs)
• Two pose model variants: RTMPose-S (fast, bundled) and HRNet-w32 (accurate, downloaded on demand)
• Truly cross-platform: compatible with Android, iOS, macOS, Windows, and Linux
• Configurable performance with XNNPACK, GPU, and CoreML acceleration

Quick Start

import 'package:animal_detection/animal_detection.dart';

final detector = AnimalDetector();
await detector.initialize();

// imageBytes is a Uint8List of encoded image data (PNG, JPG, BMP, WebP, etc.)
final animals = await detector.detect(imageBytes);
for (final animal in animals) {
  print('${animal.species} (${animal.breed}) at ${animal.boundingBox} score=${animal.score}');
  if (animal.pose != null) {
    print('Keypoints: ${animal.pose!.landmarks.length}');
  }
}

await detector.dispose();

Configuration Options

The AnimalDetector constructor accepts several configuration options:

final detector = AnimalDetector(
  poseModel: AnimalPoseModel.rtmpose,       // Pose model variant
  enablePose: true,                          // Enable body pose estimation
  cropMargin: 0.20,                          // Margin around detected body for pose crop
  detThreshold: 0.5,                         // SSD detection score threshold
  performanceConfig: const PerformanceConfig(), // Auto acceleration (default)
);

Option	Type	Default	Description
poseModel	AnimalPoseModel	rtmpose	Pose model variant
enablePose	bool	true	Whether to run pose estimation
cropMargin	double	0.20	Margin around detected body crop (0.0-1.0)
detThreshold	double	0.5	SSD detection score threshold (0.0-1.0)
performanceConfig	PerformanceConfig	auto	Interpreter hardware acceleration config

Pose Model Variants

Model	Size	Decoder	Accuracy
RTMPose-S (default)	11.6 MB	SimCC-based	Fast
HRNet-w32	54.6 MB	Heatmap-based	Most accurate

HRNet is downloaded on first use and cached locally. You can track download progress:

final detector = AnimalDetector(poseModel: AnimalPoseModel.hrnet);
await detector.initialize(
  onDownloadProgress: (model, received, total) {
    print('$model: ${(received / total * 100).toStringAsFixed(1)}%');
  },
);

Detection Result

Each detected animal is returned as an Animal object:

Field	Type	Description
boundingBox	BoundingBox	Body bounding box in absolute pixel coordinates
score	double	SSD detector confidence (0.0-1.0)
species	String?	Predicted species (e.g. "dog", "cat")
breed	String?	Predicted breed (e.g. "golden_retriever", "tabby")
speciesConfidence	double?	Species classifier confidence (0.0-1.0)
pose	AnimalPose?	Body pose keypoints (null if pose estimation disabled)
imageWidth	int	Width of the source image in pixels
imageHeight	int	Height of the source image in pixels

Supported Species

The classifier recognizes the following species and breeds (mapped from ImageNet classes):

Species	Breed Count	Examples
Dog	151	Chihuahua, Labrador, German Shepherd, Golden Retriever, Pug, Dalmatian
Cat	5	Tabby, Tiger Cat, Persian, Siamese, Egyptian
Fox	4	Red Fox, Kit Fox, Arctic Fox, Grey Fox
Bear	4	Brown Bear, American Black Bear, Polar Bear, Sloth Bear
Rabbit	3	Wood Rabbit, Hare, Angora
Cow	3	Ox, Water Buffalo, Bison
Sheep	3	Ram, Bighorn, Ibex
Deer	3	Hartebeest, Impala, Gazelle
Horse	1	Sorrel
Zebra	1	Zebra

Animals detected but not matching a known species are labeled "unknown_animal".

Requirements

• Dart 3.10+ and Flutter 3.47.5+ (dartcv4 2.3.1 needs meta 1.19.0, which older flutter_test pins rule out).

• iOS builds on Xcode 27 need an iOS 15 deployment target. Set the Runner target (and platform :ios in ios/Podfile) to 15.0, add the block below to your app's pubspec.yaml, then run flutter clean:

  hooks:
    user_defines:
      dartcv4:
        ios:
          deployment_target: '15.0'
  

  This has to live in the app: Dart only reads hook user-defines from the root package, so animal_detection cannot set it for you.

Performance

Hardware Acceleration

The package automatically selects the best acceleration strategy for each platform:

Platform	Default Delegate	Speedup	Notes
macOS	XNNPACK	2-5x	SIMD vectorization (NEON on ARM, AVX on x86)
Linux	XNNPACK	2-5x	SIMD vectorization
iOS	Metal GPU	2-4x	Hardware GPU acceleration
Android	XNNPACK	2-5x	ARM NEON SIMD acceleration
Windows	XNNPACK	2-5x	SIMD vectorization (AVX on x86)

No configuration needed, just call initialize() and you get the optimal performance for your platform.

Advanced Performance Configuration

// Auto mode (default), optimal for each platform
await detector.initialize();

// Force XNNPACK (all native platforms)
final detector = AnimalDetector(
  performanceConfig: PerformanceConfig.xnnpack(numThreads: 4),
);
await detector.initialize();

// Force GPU delegate (iOS recommended, Android experimental)
final detector = AnimalDetector(
  performanceConfig: PerformanceConfig.gpu(),
);
await detector.initialize();

// CPU-only (maximum compatibility)
final detector = AnimalDetector(
  performanceConfig: PerformanceConfig.disabled,
);
await detector.initialize();

LiteRT Next CompiledModel

CompiledModel is available through the same opt-in used by the other detection packages and remains off by default:

// Try GPU first, with verified CPU/stage fallback.
await detector.initialize(useCompiledModel: true);

// Pin CompiledModel to CPU.
await detector.initialize(
  useCompiledModel: true,
  accelerators: {Accelerator.cpu},
);

Every requested compiled graph is compared with a plain-CPU Interpreter before use. A numerically unsafe GPU graph retries on CompiledModel CPU; if that also fails, only that stage uses Interpreter. Precision.fp32 is the default.

Body Pose Keypoints (24-Point)

The pose property returns an AnimalPose object with up to 24 SuperAnimal body keypoints.

Keypoint Groups

Group	Count	Points
Neck/Throat	4	Neck base, neck end, throat base, throat end
Spine	3	Back base (withers), back middle, back end
Tail	2	Tail base, tail tip
Front legs	6	Left/right thigh, knee, paw
Back legs	6	Left/right thigh, knee, paw
Body	3	Belly bottom, body middle left/right

Accessing Keypoints

final Animal animal = animals.first;

if (animal.pose != null) {
  // Iterate through all keypoints
  for (final kp in animal.pose!.landmarks) {
    print('${kp.type.name}: (${kp.x}, ${kp.y}) confidence=${kp.confidence}');
  }

  // Access a specific keypoint
  final tail = animal.pose!.getLandmark(AnimalPoseLandmarkType.tailEnd);
  if (tail != null) {
    print('Tail tip at (${tail.x}, ${tail.y})');
  }
}

Drawing the Skeleton

Use the animalPoseConnections constant to draw skeleton lines between connected keypoints:

for (final connection in animalPoseConnections) {
  final from = animal.pose!.getLandmark(connection[0]);
  final to = animal.pose!.getLandmark(connection[1]);
  if (from != null && to != null) {
    canvas.drawLine(Offset(from.x, from.y), Offset(to.x, to.y), paint);
  }
}

Bounding Boxes

The boundingBox property returns a BoundingBox object representing the animal body bounding box in absolute pixel coordinates.

final BoundingBox boundingBox = animal.boundingBox;

// Access edges
final double left = boundingBox.left;
final double top = boundingBox.top;
final double right = boundingBox.right;
final double bottom = boundingBox.bottom;

// Calculate dimensions
final double width = boundingBox.right - boundingBox.left;
final double height = boundingBox.bottom - boundingBox.top;

print('Box: ($left, $top) to ($right, $bottom)');
print('Size: $width x $height');

Model Details

Model	Size	Input	Purpose
SSD body detector	~4 MB	320x320	Animal body detection and bounding box
Species classifier	~9 MB	224x224	Species and breed classification
RTMPose-S	11.6 MB	256x256	24-point body pose estimation (fast)
HRNet-w32	54.6 MB	256x256	24-point body pose estimation (accurate)

OpenCV Mat Input

For advanced use cases (e.g. camera frames), you can pass an OpenCV Mat directly:

import 'package:animal_detection/animal_detection.dart';

final mat = imdecode(imageBytes, IMREAD_COLOR);
final animals = await detector.detectFromMat(
  mat,
  imageWidth: mat.cols,
  imageHeight: mat.rows,
);

Live Camera Detection

For real-time detection, pass each camera package image directly to the detector. Packing happens on the caller, while color conversion, rotation, downscaling, and inference stay in the detector worker isolate.

final animals = await detector.detectFromCameraImage(
  cameraImage,
  rotation: rotation,
  isBgra: Platform.isMacOS,
  maxDim: 640,
);

For lower-level integrations, use prepareCameraFrame(...) followed by detectFromCameraFrame(...).

Background Isolates

To run detection in a background isolate, use initializeFromBuffers to avoid asset loading issues:

await detector.initializeFromBuffers(
  bodyDetectorBytes: bodyModelBytes,
  classifierBytes: classifierModelBytes,
  speciesMappingJson: speciesMappingJsonString,
  poseModelBytes: poseModelBytes,   // optional
);

HRNet Cache Management

Check if the HRNet model is already cached or clear the model cache:

final cached = await AnimalDetector.isHrnetCached();
await ModelDownloader.clearCache();  // Deletes all cached models

Error Handling

Exception	When
StateError	detect() or detectFromMat() called before initialize()
HttpException	HRNet model download fails (non-200 status)

If image decoding fails or no animals are detected, detect() returns an empty list.

Credits

The body detection and pose models are format conversions of the SuperAnimal-Quadruped checkpoints from the Mathis Laboratory, part of DeepLabCut. If you use them, please cite:

@article{ye2024superanimal,
  title={SuperAnimal pretrained pose estimation models for behavioral analysis},
  author={Ye, Shaokai and Filippova, Anastasiia and Lauer, Jessy and
          Schneider, Steffen and Vidal, Maxime and Qiu, Tian and
          Mathis, Alexander and Mathis, Mackenzie Weygandt},
  journal={Nature Communications},
  volume={15},
  number={1},
  pages={5165},
  year={2024},
  doi={10.1038/s41467-024-48792-2}
}

The species classifier is converted from torchvision's MobileNet_V3_Small_Weights.IMAGENET1K_V1.

License

The Dart source code is Apache 2.0; see LICENSE.

The bundled SuperAnimal models are an exception. assets/models/superanimal_ssdlite_float16.tflite, assets/models/superanimal_rtmpose_s_float16.tflite, and the HRNet pose model downloaded on demand are format conversions of the Mathis Laboratory's SuperAnimal checkpoints. Those weights are licensed for academic, non-commercial purposes only, and the licence is explicitly non-transferable. It also forbids using the models to deliberately harm an animal. See NOTICE.

This means using this package in a commercial product is not something this licence permits, because doing so runs those weights. The Dart code stays Apache 2.0 and can be used commercially with pose and detection models you supply yourself.

Commercial permission is not this package author's to give. The rights holders provide a route: Prof. Mackenzie W. Mathis ([e-mail]) and the EPFL Technology Transfer Office ([e-mail]).

Not affected: species_classifier_float16.tflite is BSD-3-Clause via torchvision, and species_mapping.json is this package's own work under Apache 2.0. Both are fine for commercial use.

Example

The sample code includes matching live-camera, still-image, and video-file demos. All three paint body boxes, species labels, and 24-point pose keypoints; video output uses temporal smoothing and can be replayed in the app.