Skip to main content

flutter_edge_ai_onnx 0.9.2 copy "flutter_edge_ai_onnx: ^0.9.2" to clipboard
flutter_edge_ai_onnx: ^0.9.2 copied to clipboard

ONNX Runtime engine for flutter_edge_ai: GenAI inference and embeddings on 5 native platforms; web via Transformers.js and onnxruntime-web.

flutter_edge_ai_onnx #

Renamed from flutter_gemma_onnx. Same package, new name: swap the dependency and the package:flutter_gemma_onnx/ imports; nothing on the device changes. See the migration guide.

ONNX Runtime engines for flutter_edge_ai: text generation via ORT-GenAI (OnnxEngine) and embeddings via plain ONNX Runtime (OnnxEmbeddingBackend). On native platforms (macOS, Linux, Windows, Android, iOS), both are pure dart:ffi — no JVM, no gRPC — and both drive the native library from a long-lived worker isolate so no FFI pointer ever has to cross an isolate boundary. On Web there is no FFI: generation runs through Transformers.js and embeddings run through onnxruntime-web, behind the same public API.

Teach your AI assistant this package #

dart run skills@ get --all

Installs the agent skills flutter_edge_ai bundles — this package depends on it, so they come with it. One of them, flutter-edge-ai-onnx, covers ORT-GenAI generation and ONNX embeddings, native and through Transformers.js.

Register #

Embeddings also need flutter_edge_ai_embeddings, which supplies the tokenizers this package asks core for:

flutter pub add flutter_edge_ai_embeddings
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
import 'package:flutter_edge_ai_onnx/flutter_edge_ai_onnx.dart';

await FlutterEdgeAi.initialize(
  inferenceEngines: [OnnxEngine()],
  embeddingBackends: [OnnxEmbeddingBackend()],
  embeddingTokenizers: [GemmaEmbeddingTokenizers()],
);

Either arm can be registered on its own — they don't depend on each other.

Since 0.5.8 this package needs flutter_edge_ai 2.3.0, and 2.3.0 needs every flutter_edge_ai package upgraded together: flutter_edge_ai_litertlm 1.13.0 · flutter_edge_ai_embeddings 2.3.0 · flutter_edge_ai_mediapipe 1.2.2 · flutter_edge_ai_builtin_ai 0.3.5 · flutter_edge_ai_speech 0.5.6 · flutter_edge_ai_agent 0.2.9 · flutter_edge_ai_rag 1.1.0 · flutter_edge_ai_sqlite 2.1.0 · flutter_edge_ai_qdrant 2.1.0 · genkit_flutter_edge_ai 0.9.1. Run flutter pub upgrade; the migration guide lists the API changes.

Platform matrix #

Platform Native archives OnnxEngine/OnnxEmbeddingBackend enabled
macOS (Apple Silicon) ✅ bundled ✅ device-verified
macOS (Intel) ❌ ❌
Linux x64 ✅ bundled ✅ device-verified
Windows x64 ✅ bundled ✅ device-verified
Windows ARM64 ✅ bundled ✅ verified on a Windows 11 ARM64 runner
Android (arm64) ✅ bundled ✅ device-verified
iOS (arm64) ✅ bundled ✅ device-verified
Web N/A — Transformers.js + onnxruntime-web, no native archive ✅ both arms

OnnxEngine.canHandle/OnnxEmbeddingBackend.describe are gated to macOS arm64, Linux x64, Windows x64 and ARM64, Android arm64, and iOS arm64 — device-verified end-to-end (generation + embeddings) on macOS (~54 tok/s, M4 Pro), Linux (~5.3-5.8 tok/s), Windows x64 (~3.3 tok/s), and Android (FTL Pixel 8 Pro, ~10.4 tok/s, ~3.74 GB RSS for a 3.8B int4 model). On iOS the framework-embedding/dlopen path builds, signs, installs and launches on a real iPhone, and generation runs (the @executable_path-anchored dlopen resolves the single self-contained genai xcframework — the same proven pattern as flutter_edge_ai_litertlm's iOS path). Windows ARM64 uses the same FFI path and checksum-pinned Microsoft archives. On a GitHub-hosted Windows 11 ARM64 runner (Azure Cobalt 100), the example built for ARM64, generation passed 2/2 (64 tokens at 10.9 tok/s), and MiniLM embeddings passed (dimension 384). The bundled DLLs are ARM64 builds. On an unsupported native host (macOS Intel, or any other native ABI) OnnxEngine politely declines (logs why, lets another registered engine — or core's own "no engine can handle this" error — take over) instead of dlopen-ing a library the app may or may not have bundled. Web has no dlopen step at all — OnnxEngine/OnnxEmbeddingBackend run there via the Transformers.js/onnxruntime-web arms below instead. OnnxEmbeddingBackend.canHandle stays extension-based on every platform for a different reason (so a catch-all embedding backend like LiteRtEmbeddingBackend never silently claims an .onnx/.ort file); its platform gate lives in describe instead, as a loud EmbeddingLoadException.

Android needs minSdk 24 — ORT's Android AAR declares minSdkVersion=24; raise your app's android/app/build.gradle(.kts) minSdk to 24 or higher if it's lower today. The device-verified Android model (Phi-3.5-mini 3.8B int4) peaks at ~3.74 GB RSS — plan for 6 GB+ RAM devices; smaller models scale down.

Which output the embedding path reads #

A session's outputs are preferred in the order sentence_embedding > pooler_output > last_hidden_state > first declared. The first two are already-pooled [1, dim] sentence embeddings and are copied verbatim; only last_hidden_state is per-token and gets mean-pooled. A graph exposing both pooler_output and last_hidden_state — SigLIP2's text tower does — would otherwise fall through to the per-token output and be pooled a second time.

This is a heuristic on a name, and it is worth knowing where it stops being one. Optimum feature-extraction exports (sentence-transformers/, Xenova/ MiniLM and friends) declare no pooler_output at all, so they are unaffected. A graph produced by exporting BertModel directly does declare one — BERT's tanh NSP pooler — and this preference will now read that instead of mean-pooling the hidden states. That head is not a sentence embedding and is not L2-normalized; if you export a BERT encoder yourself, export it for feature extraction, or strip the pooler. Which output to read is really a property of the model profile, not of the output's name, and should move there.

Inference — OnnxEngine #

Text-only (v1): no vision, no audio, no LoRA. openSession / openChat open independent sessions on the same loaded model (on web, see below). Each holds its own ORT-GenAI generator, so its context is its own and closing it frees it; turns of different sessions run one at a time, and nothing is replayed. Cost: a generator allocates its whole KV cache at its first turn, maxTokens × the cache size per token (2 × layers × KV heads × head size × dtype size; 0.75 MiB for Phi-3.5-mini int4, which the figures below are for), resident from then on (Phi-3.5-mini int4 on Linux CPU: 771 MiB at 1024, 1539 at 2048, 3077 at 4096). On Windows closing a session gives the whole amount back (commit -383 MiB at 512, -768 MiB at 1024). On Linux the process's RSS did not go down at 512, 1024 and 2048 (it did at 4096), so there budget the peak, not the count of open sessions. Opening more than maxConcurrentSessions throws a StateError; unset, that is 2 on Android and iOS and unlimited on desktop. An unset cap cannot mean "unlimited" on a phone: pass a larger maxConcurrentSessions to allow more. Sampling follows the model's genai_config.json (greedy unless it sets do_sample); the temperature, topK, topP and randomSeed you pass override it, and temperature: 0 or topK: 1 keeps decoding greedy. Natively, a temperature or topP without topK over a config with top_k: 1 also gets a top-k (the family's, else 40), so it samples. On web the seed is not applied yet and topP is ignored (Transformers.js samples with top-k only); each is said once when set. Models install as ModelFileType.onnx and use ORT-GenAI's own chat template (OgaTokenizerApplyChatTemplate) — the engine never builds turn markers itself.

Response formats. openSession / openChat(responseFormat:) are enforced by ORT-GenAI's guidance (llguidance): the session's JSON schema is set on its own generator, so every complete reply is a value of the format, and core checks each one. The whole ResponseFormat.dialect is enforced on macOS arm64, for a model directory with a tokenizer.json (guidance reads it). Linux x64, Windows x64 and Windows ARM64 also ship an ORT-GenAI 0.17.0 build with guidance, and each is enabled once its run has passed; until then they refuse a format, saying so. Android (this repository's ORT-GenAI build), iOS (Microsoft's) and the web have no guidance and refuse it with ResponseFormatUnsupportedException; read model.structuredOutputSupport first.

After a turn that does not end on its own, the session stays usable with its history, and a turn that streamed text stays in the model's context exactly as core's InferenceChat keeps it in the chat. The worker settles the turn before it serves the next one:

  • A stop and a guided turn at maxOutputTokens keep the exchange: the prompt and the reply as far as the user saw it stay in the context, closed with the model's end-of-turn token, as if the reply had ended there (LiteRT-LM does the same). A stop before any text keeps nothing, as core's chat takes it back; a guided turn at its cap keeps its prompt even then.
  • A window filled during decode has no room for that token and keeps the turn as it is. The window is then full, so a session used directly refuses every later prompt with ContextLimitException: close it and open a new one, or use InferenceChat, which rebuilds its session from its history.
  • A failure and a close are rewound to where the generator was before the turn (OgaGenerator_RewindTo): the earlier turns stay, this one goes.
  • An unguided turn at its cap is kept and continued, as before.

A guided session's grammar starts again after each of these. A prompt that does not fit is refused and leaves the conversation as it was: on a plain and a structured session alike the reply fails with core's ContextLimitException and the turn ends TurnEnd.contextLimit, the signal for an input that was refused.

Measured on Phi-3.5-mini (macOS, greedy):

  • After a guided turn cut at its cap (a title, {"answer": "Whispersings), asked which title it had started, the model answers from a context that holds that turn — "Tempest's Symphony: Ode to the Storm's Fury and Grace", where the same conversation written in as text gives "…Odes to…", and the turn dropped gives an unrelated title.
  • A story that fills the window ends contextLimit; the next question is refused with ContextLimitException, and again when asked twice.
  • After a stop, asked for the story's first words, the model quotes the partial reply ("Once upon"), the same as from the conversation written as text with the partial reply in it; with the turn dropped it has no story to quote.
  • Longer continuations of those two contexts can drift apart: ORT-GenAI never computes a finished turn's end token, and the chat template writes a newline after it, so they are not token-identical.

A model ORT-GenAI cannot rewind (phi3v, whisper, LFM2, decoder pipelines) gets its context rebuilt from the session's own conversation text instead, at the cost of a full prefill, with one printed warning per model.

Natively activeBackend is null: ORT-GenAI picks the execution provider from the model's genai_config.json, whose execution-provider section the plugin does not read, so there is no honest answer to report. On web it is null until the first openSession loads the Transformers.js pipeline, then gpu (WebGPU) or cpu (WASM).

Model layout: a directory, not a file #

An ORT-GenAI model is a directory, not a single file:

my-model/
├── genai_config.json
├── model.onnx (+ model.onnx_data for the external-weights case)
└── tokenizer files (tokenizer.json, tokenizer_config.json, …)

OnnxEngine.createModel takes that directory's genai_config.json and loads the parent directory as the model. There are two ways to get the directory onto the device.

From a Hugging Face repo (one call). OnnxHuggingFaceResolver lists the repo's file tree, picks an execution-provider folder, and installs every file in it into a per-model subdirectory — so fromHuggingFace(repo) downloads the whole ORT-GenAI bundle. The resolver rides on OnnxEngine via HuggingFaceResolverSource, so registering the engine is enough:

await FlutterEdgeAi.initialize(inferenceEngines: [OnnxEngine()]);

final install = await FlutterEdgeAi.installModel(
  // ONNX repos declare no model family — the caller's modelType is used as-is.
  modelType: ModelType.general,
  fileType: ModelFileType.onnx, // selects the ONNX resolver
).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install();

A repo that ships several EP variants (cpu_and_mobile/…, cuda/…) resolves to a CPU/mobile folder automatically — the bundled ORT-GenAI runtime is CPU-only, so a CPU/mobile folder is preferred over any GPU export. (A repo that ships only GPU variants still falls back to its GPU folder — flagged in install.notes, which records the chosen folder either way.) Pin a specific folder with OnnxHuggingFaceResolver(variant: 'cpu_and_mobile/cpu-int4-…') passed to initialize(huggingFaceResolvers: [...]); a pinned GPU variant is installed but flagged in notes as one the CPU-only runtime may fail to load.

From a local directory. If you ship the bundle yourself (an asset, or a directory you pre-populate), point fromFile at its genai_config.json:

await FlutterEdgeAi.installModel(
  modelType: ModelType.general,
  fileType: ModelFileType.onnx,
).fromFile('/path/to/my-model/genai_config.json').install();

Either way, pointing modelPath at a directory missing genai_config.json fails loudly with a StateError naming the gap, rather than a confusing native error.

ORT_LIB_PATH — you don't need to configure anything #

ORT-GenAI resolves the plain ONNX Runtime library at its own native InitApi() time via a bare-name dlopen("libonnxruntime.dylib"), with a dladdr-based "look next to my own binary" fallback. Under Flutter's Native Assets .framework wrapping, onnxruntime and onnxruntime-genai land in two separate .framework bundles, so that fallback can't find it. Before opening the GenAI library, GenAiFfiClient dladdr-resolves the already-open ORT library's real on-disk path and exports it as the ORT_LIB_PATH environment variable — GenAI's own sanctioned override, checked first inside InitApi(). This needs no Podfile step, no packaging change, no app-level configuration; it just works.

For host tests / local dev, FLUTTER_GEMMA_ORT_GENAI_LIBS (a directory containing both platform-default-named dylibs) bypasses the CodeAsset bundle entirely — see test/onnx_generation_host_smoke_test.dart.

Telemetry is off, and the Windows runtime #

ORT-GenAI 0.17.0 ships Microsoft's usage telemetry on every native host, on by default (0.14.0 had none). GenAiFfiClient sets ORT_DISABLE_TELEMETRY=1 in the process before it opens the library, and keeps a value your app already set; to turn it on, set ORT_DISABLE_TELEMETRY=0 before the first model loads.

The 0.17.0 Windows DLLs import MSVCP140_ATOMIC_WAIT.dll, which the Visual C++ 2015-2022 redistributable provides. An app that ships the CRT app-local has to include that file too.

On web (Transformers.js) #

On Web, OnnxEngine doesn't drive native ORT-GenAI at all — it runs the model through Transformers.js v4 in the browser. The model identity is a Hugging Face repo id (e.g. onnx-community/Qwen2.5-0.5B-Instruct), not a directory, and install is fileless: ModelFileType.onnx just marks the repo id active — core never downloads model bytes — and Transformers.js fetches and caches the repo itself the first time you run inference. Each call runs a stateless pipeline() and resends the full chat history (Transformers.js has no persistent session), formatted with the model's own chat_template. PreferredBackend.cpu pins WASM; anything else tries WebGPU first and falls back to WASM. Same constraints as native — text-only, no vision, no audio, no LoRA. openSession / openChat sessions each keep their own history on the one pipeline, and their turns run one at a time. An open session holds no KV cache, only its history, so maxConcurrentSessions is unlimited unless you set it. See Web setup for the required web/index.html shim.

Embeddings — OnnxEmbeddingBackend #

A plain ONNX Runtime forward pass (no ORT-GenAI, no text generation) over one .onnx/.ort embedding model file and its tokenizer — not a directory, unlike the generation arm. One factory handles both:

  • WordPiece / BERT-style models (e.g. all-MiniLM-L6-v2) — tokenLevel output contract, mean-pooled + normalized client-side.
  • SentencePiece models (e.g. EmbeddingGemma-300M-ONNX) — pooledFinal output contract (the model's own sentence_embedding output).

The output contract and mask/token_type_ids requirements are discovered from the session's actual graph once it opens — no per-model configuration needed. Priority 10 (above LiteRtEmbeddingBackend's catch-all priority 0), so registering both and installing an .onnx/.ort model (any letter case) routes here.

Limits:

  • Text only. getActiveEmbedder(supportImage: true) or supportAudio: true fails with EmbeddingLoadException.
  • Input length. maxInputTokens is the graph's static sequence length, or null for a dynamic-shape graph. A static graph is padded, never truncated: a longer input — task prefix and special tokens included — fails with EmbeddingInputTooLongException.
  • Task prefixes. embed(…, taskType:) prefixes the text exactly as the LiteRT backend does; TaskType.none adds no prefix.

The pass itself is OnnxEmbeddingPass (with OnnxEmbeddingConfig), exported on native for tests and device gates that drive it directly; apps get it through FlutterEdgeAi.getActiveEmbedder().

On native the embedder reports activeBackend == cpu: no execution provider is appended, so plain ORT runs on CPU and preferredBackend is not applied. On web it is null.

For host tests / local dev, FLUTTER_GEMMA_ORT_LIBRARY overrides the resolved ORT library path directly (mirrors the inference arm's FLUTTER_GEMMA_ORT_GENAI_LIBS).

OnnxEmbeddingBackend also runs on Web, via onnxruntime-web (WebGPU/WASM) instead of the native FFI client. This release supports WordPiece/BERT-style embedding models such as all-MiniLM-L6-v2 on Web; SentencePiece models such as EmbeddingGemma-300M-ONNX are native-only. See Web setup.

What v1 does not do #

  • No vision, no audio, no multimodal input on the inference arm — text only.
  • No LoRA.
  • openSession / openChat sessions are independent and must be closed by their caller (see Inference above).

Web setup #

Web needs a small web/index.html shim before FlutterEdgeAi.initialize() runs — the same readiness-handshake pattern flutter_edge_ai_litertlm uses for @litert-lm/core. Add the shim for whichever arm(s) you register:

<!-- Transformers.js v4 (OnnxEngine web generation). -->
<script type="module">
window.transformersReady = (async () => {
  const m = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1');
  window.transformers = m;
  return m;
})();
</script>

<!-- onnxruntime-web (OnnxEmbeddingBackend web embeddings). -->
<script type="module">
window.ortReady = (async () => {
  const m = await import('https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/ort.bundle.min.mjs');
  m.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/';
  window.ort = m;
  return m;
})();
</script>

Dart awaits window.transformersReady / window.ortReady before touching either module's dart:js_interop bindings, so the shim must run before the Flutter app boots (i.e. in <head>, ahead of flutter_bootstrap.js).

See also #

0
likes
140
points
1.26k
downloads

Documentation

API reference

Publisher

verified publishersashadenisov.dev

Weekly Downloads

ONNX Runtime engine for flutter_edge_ai: GenAI inference and embeddings on 5 native platforms; web via Transformers.js and onnxruntime-web.

Homepage
Repository (GitHub)
View/report issues

Topics

#gemma #llm #onnx #onnxruntime #on-device

License

MIT (license)

Dependencies

code_assets, crypto, ffi, flutter, flutter_edge_ai, hooks, mutex

More

Packages that depend on flutter_edge_ai_onnx