edge_vision_core 0.1.0
edge_vision_core: ^0.1.0 copied to clipboard
Deterministic C++17 image & camera frame preprocessor, EXIF orienter, letterbox resizer, tensor normalizer, and YOLO postprocessor for Flutter Edge AI.
edge_vision_core #
Deterministic C++17 image & live camera frame preprocessor, EXIF orienter, letterbox resizer, float32 tensor normalizer, and YOLO post-processor for Flutter Edge AI (Android, iOS, Windows, macOS, Linux).
Built around a lightweight, zero-external-dependency C++17 core with a pure extern "C" ABI (src/edge_vision_core.h) so mobile apps produce identical pixel math across iOS, Android, Desktop, and training pipelines.
Why edge_vision_core? #
| Cross-Platform Challenge | What Fails with OS Native APIs | Solution in edge_vision_core |
|---|---|---|
| iOS vs. Android Accuracy Drift | UIImage / CoreGraphics and Android Bitmap / Skia use different resize filters and color conversions. |
One shared C++17 resizing & normalization pipeline across every platform. |
| Broken EXIF Orientation | Portrait photos rotated via EXIF tags (1..8) end up sideways or mirrored in inference tensors. |
Parses JPEG APP1 TIFF/IFD0 tags (0x0112) and physically rotates/flips pixels before resize. |
| YOLO Letterbox Coordinate Drift | Unscaling 640x640 detections back to 1920x1080 requires tracking exact scale and padding offsets. |
Records TransformMeta (scale, padX, padY, oriented dims) and unscales detections in C++. |
| Slow Pure-Dart Decoding | Pure-Dart image packages block or take seconds on 12MP–48MP photos. | Native C++ decoding & packing (~2–5 ms resize/pack), with preprocessInIsolate support. |
| Heavy Binary Bloat | Bundling OpenCV adds 15–30 MB to mobile app binaries. | Lightweight header-only C++ core (stb_image, stb_image_resize, stb_image_write). |
Architecture #
[ Flutter (dart:ffi) ] [ React Native (JSI/Nitro) ] [ Python / CMake ]
\ | /
+--------------------------+-------------------------+
|
[ C ABI: src/edge_vision_core.h ]
|
[ Pure C++17 Vision Core (src/) ]
1. Decode / YUV420->RGB -> 2. EXIF 1..8 Orient -> 3. Letterbox / Crop
4. NCHW / NHWC Float32 -> 5. EXIF-free JPEG -> 6. YOLO Postprocess + NMS
Features #
- Encoded Image Decoding & EXIF Normalization: JPEG, PNG, BMP, GIF, TGA, PSD, HDR, PNM decoding with automatic JPEG EXIF orientation (
1..8) physical correction. - Deterministic Resizing:
ResizeMode.letterbox(standard YOLO 114 gray border),ResizeMode.stretch, andResizeMode.centerCrop. - Single-Pass Tensor Packing:
RGB/BGR,NCHW([1, 3, H, W]) /NHWC([1, H, W, 3]), customscale,mean,stdDev, plus ready-to-use presets:PreprocessConfig.yolo(size: 640)PreprocessConfig.imageNet(width: 224, height: 224)PreprocessConfig.clip(size: 224)
- Live Camera YUV420 / NV12 / NV21 Support:
preprocessYuv420(...)with deterministic BT.601 conversion and clockwise sensor rotation (0,90,180,270). - EXIF-Free Cloud Upload Optimization:
resizeAndEncodeJpeg(...)shrinks high-res photos deterministically and strips metadata before sending to Vision LLMs or REST APIs. - YOLO Post-Processing & Unscaling:
postprocessYolo(...)supports both end-to-end NMS-free outputs ([1, N, 6], e.g. YOLO26 / YOLOv10) and classic transposed outputs ([1, 4 + C, N]) with class-aware Greedy NMS, returning coordinates mapped back to the oriented original image. - Concurrency Safe: Serializes concurrent calls on reusable native scratch buffers (
Future.waitsafe) and providesVisionPreprocessor.preprocessInIsolate(...)for heavy photos.
Getting Started #
Add edge_vision_core to your pubspec.yaml:
dependencies:
edge_vision_core: ^0.1.0
1. End-to-End YOLO Preprocessing & Postprocessing #
import 'dart:typed_data';
import 'package:edge_vision_core/edge_vision_core.dart';
Future<void> runYoloPipeline(Uint8List photoBytes) async {
final vision = VisionPreprocessor();
try {
// 1. Decode, apply EXIF orientation, letterbox to 640x640, normalize [0, 1] NCHW
final prep = await vision.preprocessBytes(
photoBytes,
PreprocessConfig.yolo(size: 640, layout: TensorLayout.nchw),
);
// 2. Feed `prep.tensor` (Float32List) into ONNX Runtime / TFLite / ncnn
final Float32List rawModelOutput = await myRuntime.run(prep.tensor, prep.shape);
// 3. Post-process detections & unscale back to original oriented image coordinates
final detections = vision.postprocessYolo(
rawModelOutput,
numClasses: 80,
numAnchors: 300,
isEndToEnd: true, // true for [1, N, 6] NMS-free YOLO, false for [1, 4+C, N]
confThreshold: 0.35,
iouThreshold: 0.45,
meta: prep.meta,
);
for (final det in detections) {
print('class=${det.classId} conf=${det.confidence.toStringAsFixed(2)} '
'box=[${det.box.x1}, ${det.box.y1}, ${det.box.x2}, ${det.box.y2}]');
}
} finally {
vision.dispose();
}
}
2. Live Camera Frame (YUV420 / NV12) Preprocessing #
final prep = await vision.preprocessYuv420(
yPlane: cameraFrame.yBytes,
yStride: cameraFrame.yRowStride,
uPlane: cameraFrame.uBytes,
uStride: cameraFrame.uvRowStride,
vPlane: cameraFrame.vBytes,
vStride: cameraFrame.uvRowStride,
uvPixelStride: cameraFrame.uvPixelStride, // 1 for planar I420, 2 for NV12/NV21
width: cameraFrame.width,
height: cameraFrame.height,
rotationDegrees: 90,
config: PreprocessConfig.yolo(size: 640),
);
3. Deterministic EXIF-Free JPEG Downscaling for Vision APIs #
final encoded = await vision.resizeAndEncodeJpeg(
highResPhotoBytes,
targetWidth: 1024,
targetHeight: 1024,
resizeMode: ResizeMode.letterbox,
quality: 88,
);
// Send `encoded.bytes` cleanly over HTTP with EXIF orientation baked in & stripped.
Reusing the C++ Core Outside Flutter #
The pure C++17 core lives entirely inside src/ (edge_vision_core.h, edge_vision_core.cpp, exif.cpp, CMakeLists.txt). You can build it directly via CMake for React Native (JSI / Nitro Modules), Python (ctypes), or C++ servers:
cmake -S src -B build/cpp_test -DEVC_BUILD_TESTS=ON
cmake --build build/cpp_test --config Release
ctest --test-dir build/cpp_test -C Release --output-on-failure