mediapipe_core 0.1.3
mediapipe_core: ^0.1.3 copied to clipboard
Shared layer for the MediaPipe task packages: Google's engine on each platform, verified downloads, an on-demand model store and the browser runtime.
mediapipe_core #
The shared layer of the MediaPipe task packages (mediapipe_vision,
mediapipe_text, mediapipe_audio, mediapipe_decision and
mediapipe_retrieval); apps normally get it through them rather than depending
on it directly. It owns everything the task families share, so each of vision, text and audio works alone and an app
using several loads one copy of each shared piece. That includes the code that
downloads and verifies Google's MediaPipe libraries, which each family's build
hook uses to bundle its own:
| Platform | Runtime |
|---|---|
| Android 9+ (arm64-v8a, x86_64; armeabi-v7a on the CPU, without the large models), iOS 15+ (device, arm64 simulator), macOS 14+ arm64, Linux x64, Windows x64 | Google's MediaPipe 1.2.0 C library for each family (MediaPipeTasksVisionC, libmediapipe_tasks_text, and so on), bundled by that family's hook |
| Web | Google's JavaScript runtime, through each family's web plugin |
An app ships only the libraries of the families it depends on, and needs no
setting to get them. The hooks download them from a pre-release of
hugocornellier/mediapipe_flutter_native, where Google's development builds
are posted unmodified until Google publishes them, and check every file's
SHA-256 and size (lib/src/native_assets/family_runtimes.dart).
hooks.user_defines.mediapipe_core.asset_source can point at a directory
holding them, named by SHA-256, instead.
Installation #
An app using a family does not need to add this package: it comes with
flutter pub add mediapipe_vision or any other family. That covers its types
and its commands, dart run mediapipe_core:bundle_models and
dart run mediapipe_core:web_runtime; use dart run, since the older
flutter pub run refuses a package your app does not list itself.
Add it from pub.dev only if your own code imports package:mediapipe_core/...,
because the depend_on_referenced_packages lint in Flutter's default rules
expects an imported package to be listed:
flutter pub add mediapipe_core
What every family shares #
Each family's library (package:mediapipe_vision/mediapipe_vision.dart,
mediapipe_text, mediapipe_audio, mediapipe_decision and
mediapipe_retrieval) re-exports
package:mediapipe_core/mediapipe_core.dart, so these types are the same
on every platform and in every family:
TaskOptions, the base of every options class: exactly one ofmodel(a pinnedDownloadAsset),modelPathandmodelBytes, plusdelegate(Delegate.cpuorDelegate.gpu).- The value types results are made of, named after Google's containers:
MediaPipeCategory,Classifications,Embedding,Detection,BoundingBox,NormalizedKeypoint,NormalizedLandmark,Landmark,Matrix,ConfidenceMaskandCategoryMask. MediaPipeExceptionand its subtypes:RuntimeUnavailableException(with afix),ModelDownloadExceptionandTaskException(Google's runtime refused a call;statusCode,gpuUnavailable).TaskCapabilitiesandTaskPlatform, which every task'squeryXxxCapabilities()returns;ModelStoreandModelSource; andMediaPipeWebRuntime.
package:mediapipe_core/platform_interface.dart holds what family packages
and platform plugins need and apps do not.
Verified downloads and offline builds #
Every native runtime and model pin names a primary URL, optional published mirrors, and one SHA-256. Core tries URLs in order and checks the digest before publishing a file. It reports every failed location when none matches. Each native library's primary URL names its file inside Google's nightly archive, which core reads with HTTP range requests, and its mirror is a copy in a GitHub pre-release. Model pins name no mirrors.
Each request gets three attempts on a connection error, a timeout or HTTP
408, 429, 500, 502, 503 or 504, waiting a random backoff or the host's
Retry-After (at most 30 seconds), and a dropped download resumes where it
stopped. At the last URL, a failed lookup or a lost connection keeps being
retried for 75 seconds from the first failure, so a short network outage
passes and a device with no network sees the error after that long. Any other
failure, including a digest mismatch, moves to the next URL at once.
For offline or mirrored builds, point every download at a local directory or an internal HTTP(S) root holding files named by their full SHA-256. That source is then used exclusively: the digest check still applies, and a missing or wrong file fails the build rather than falling back to the internet.
-
Build hooks read it from the app's pubspec (a relative directory resolves against that pubspec):
hooks: user_defines: mediapipe_core: asset_source: third_party/mediapipe-assetsHooks cannot use an environment variable here: Dart's hook runner passes them only an allowlist of variables.
-
ModelStore(source: ...)takes the same kind of source at run time. -
Command-line tools (
dart run) read theMEDIAPIPE_ASSET_SOURCEenvironment variable.
From a checkout of this repository, fill such a directory for the build targets you need. Models are included for every target, since apps may use any family:
python3 -B packages/mediapipe-core/tool/mirror_runtime_assets.py \
--target macos/arm64 --target ios/arm64 --prefill /path/to/mediapipe-assets
The same tool without --prefill is a dry run inventory. It lists each pin,
its target, license information when known, and a proposed GitHub release
asset name. It never uploads anything. The prefill path uses core's verified
downloader. Copy the resulting SHA-named directory to the build machine, or
serve it from an internal URL root.
Bundled models #
Apps bundle the models they use at build time. Each family's XxxModels.byName
names its models. The app lists the ones it needs per family in its pubspec,
declares assets/mediapipe/ under flutter: assets:, and runs from its root:
dart run mediapipe_core:bundle_models
The command downloads each listed model (from asset_source when it is set),
checks it against its pinned SHA-256, and writes it into assets/mediapipe/,
named by that SHA-256 with a readable manifest.json beside it. Models that
are no longer listed are removed. With --check it changes nothing and fails
when the folder does not match the lists, for CI. It reads each family's
models by running a small generated program,
.dart_tool/mediapipe_core/bundle_models_registry.dart, with the app's
packages, since core cannot depend on the families.
A task created with model: then looks for its model in this order:
- the store's cache;
- the app's bundled copy, copied into the cache once on native platforms, where MediaPipe takes a file path;
- a download, only when
ModelStore.allowDownloadsis true.
Otherwise create throws a RuntimeUnavailableException whose fix names
the pubspec entry to add. ModelStore().find(model) performs the same lookup
without ever using the network.
On-demand models #
Downloading at run time is opt-in. Set ModelStore.allowDownloads = true
before creating tasks to let model: download what the app does not bundle;
calling get or prefetch directly always may download.
ModelStore and DownloadAsset come with every family's library. A family
model pin can be passed directly to get or prefetch; clear removes the
store's entries. get and find return a ModelSource: on native platforms
its path is a file in application support, which can be given to the task
options as modelPath. Each model keeps its original file name. Downloads go to an adjacent
temporary file under a per-model lock shared across isolates and processes,
and cached bytes are checked again on every read. iOS excludes this model folder
from iCloud backup. On web get returns verified bytes from Cache Storage
or the app's bundle, and otherwise downloads them. An invalid cached
response is discarded.
Core uses Flutter's path_provider for application support rather than
asking every family to supply a directory, so the storage policy stays shared.
import 'package:mediapipe_text/mediapipe_text.dart';
Future<void> prepareModels() async {
final store = ModelStore();
await store.prefetch(TextModels.bertClassifier);
final source = await store.get(TextModels.bertClassifier);
print(source.path); // A file on native platforms; `bytes` in browsers.
}
Downloading needs Android's INTERNET permission, and a sandboxed macOS app
needs com.apple.security.network.client. Bundled models and already
verified cache entries work without network access.
Web runtime #
Browsers load Google's pinned MediaPipe JavaScript and WASM runtime for each
task family (vision, text, audio) from jsDelivr. One setting moves all of them.
For offline use or a strict Content Security Policy, copy the verified
runtimes of the families your app uses into its web/ folder, from the app
root:
dart run mediapipe_core:web_runtime web/mediapipe
The tool downloads each family's pinned npm archive, checks its integrity and each listed file's SHA-384, and writes only the files the browser loads. Workers verify those same file digests from any configured root before loading the bundle and WASM through Blob URLs. Then, before creating the first task:
import 'package:mediapipe_vision/mediapipe_vision.dart';
void useSelfHostedRuntime() {
MediaPipeWebRuntime.baseUrl = 'mediapipe/';
}
baseUrl is an npm-style root, so another npm CDN such as
https://unpkg.com/ works as well. Each family's library exports the same
class, and it does nothing off the web.
Serve the files with CORS and CSP rules appropriate to your origin.
Issues and feedback #
Please file issues, bugs, or feature requests in our issue tracker.
Issues that are specific to Flutter can be filed in the Flutter issue tracker.
To contribute a change to this plugin, please review our contribution guide and open a pull request.