flutter_pixelmatching 0.1.6

Module to perform feature matching using OpenCV

Flutter PixelMatching

Plugin to perform feature matching using OpenCV

An image comparison module written as an exercise in Flutter FFI and out of a need for that functionality. It is called PixelMatching, but it is actually a FeatureMatching module utilizing OpenCV. Isolate was utilized for asynchronous processing. For the internal functionality, it was written in C++.

The matching algorithm is using FLannBasedMatcher, and for the detector, Andorid uses SIFT iOS uses KAZE.

There is currently no method to change the algorithm, so if you want to change the algorithm and test it, please refer to ios/Classes/src/ImageProcessor.cpp and change it.

ImageProcessor::ImageProcessor() : stateCode(NotInitialized) {
    compare.setMatchers(cv::DescriptorMatcher::create(DescriptorMatcher::MatcherType::FLANNBASED));
#ifdef __ANDROID__
    compare.setDetector(cv::SIFT::create());
#elif __APPLE__
    compare.setDetector(cv::KAZE::create());
#endif
}

Demo Screenshots

Features

• Plugins utilizing OpenCV
• Perform image comparisons on camera streams (mobile platforms only), image files.
• Compares image features to calculate and return similarity

Todo

• Write plugin test code
• Reduce the overall size of the plugin
  • Do not build x86 Flutter Building the app for release
  • Find a way to avoid importing third-party library sources directly inside the plugin

Supported platforms

Flutter PixelMatching is available for Android and iOS.

• Android (min SDK 24)
• iOS (min iOS version 11.0)

flutter_pixelmatching is currently written to be available only on mobile and there are no plans for other platforms.

Getting started

Installing dependencies

flutter pub add flutter_pixelmatching

or

dependencies:
  flutter_pixelmatching: ^0.1.5

Usage

final matching = PixelMatching();

// An enum value indicating the processing status of the module. If a marker is registered after initialization, it will change to the 'noQuery' state. 
enum PixelMatchingState {
  // The pixel matching is not initialized.
  notInitialized,
  // target image is not set.
  noMarker,
  // query image is not set.
  noQuery,
  // is processing.
  processing,
}

Initialize

Initialize while registering the marker image.
Support ImageType jpeg/bgra8888
For ease of image handling, we currently support two image types.

// from Camera Stream 
final imgType = Platform.isAndroid ? "jpeg" : "bgra8888";
final img = cameraImage.planes[0].bytes;
final w = cameraImage.width;
final h = cameraImage.height;
matching?.initialize(imgType, img, w, h);

// from ImagePicker
final imagePicker = ImagePicker();
final xImage = await imagePicker.pickImage(source: ImageSource.camera);
if (xImage != null) {
  final image = await imglib.decodeJpgFile(xImage.path);
  matching?.initialize("jpeg", imglib.encodeJpg(image!), image.width, image.height);
}

Query

Calculates the current similarity compared to the registered image at initialization.

final status = await matching.getStateCode();
if(status == PixelMatchingState.noQuery) {
  final imgType = Platform.isAndroid ? "jpeg" : "bgra8888";
  final img = cameraImage.planes[0].bytes;
  final w = cameraImage.width;
  final h = cameraImage.height;
  final confidence = await matching?.setQuery(imgType, img, w, h, rotation: rotation);
  print(confidence);
  setState(() {
    similarity = confidence ?? 0.0;
  });
}

MarkerQueryDifferenceImage

Functions to visually see which images are currently being processed. This is an optional function that is provided separately to ensure that images are being processed well during development.

// Use package:image for functional convenience
imglib.Image? preview = await matching?.getMarkerQueryDifferenceImage();
if (preview != null) {
  setState(() {
    previewImage = imglib.encodeJpg(preview);
  });
}

⚠️ Issue

Currently, for ease of use, the libopencv_java4.so binary file is included inside the plugin. I know it's not recommended, but it's there so you can try it out quickly. If you are using the OpenCV module in another plugin, it may cause conflicts. Please note that.

License

MIT License. See LICENSE for details.