flutter_gemma_onnx library

ONNX Runtime on-device engines for flutter_gemma: text generation (OnnxEngine, native macOS/Linux/Windows/Android/iOS arm via ORT-GenAI PLUS a web arm via Transformers.js, see below) and embeddings (OnnxEmbeddingBackend, productionized on native + web).

Opt-in. Add to pubspec.yaml and pass instances to FlutterGemma.initialize(...).

Inference (OnnxEngine) — see OnnxEngine's doc comment.

• Native (macOS/Linux/Windows/Android/iOS): text-only, greedy decoding, one session at a time, over ORT-GenAI via dart:ffi (GenAiFfiClient, a long-lived worker isolate — no FFI handle ever crosses an isolate boundary).
• Web: text generation via Transformers.js v4 (@huggingface/transformers) — a different model-provisioning + execution story than the native arm: the model is identified by its Hugging Face repo id (not an ORT-GenAI directory), Transformers.js resolves + caches the repo itself, and the pipeline is stateless per call (the session resends the whole chat history every turn). See OnnxWebInferenceModel's module doc (lib/src/web/) for the details.

Embeddings (OnnxEmbeddingBackend) — a plain ONNX Runtime forward pass (no ORT GenAI) over an .onnx/.ort embedding model directory. Native: dart:ffi, both MiniLM-family (WordPiece) and EmbeddingGemma-300M-ONNX (SentencePiece) route through one factory, see OnnxEmbeddingBackend's doc comment. Web: onnxruntime-web (WebGPU/WASM), WordPiece (MiniLM-family) only in this release — SentencePiece needs a pure-Dart parser that doesn't exist yet (native's dart_sentencepiece_tokenizer is dart:io/dart:isolate-only).

import 'package:flutter_gemma/flutter_gemma.dart';
import 'package:flutter_gemma_onnx/flutter_gemma_onnx.dart';

await FlutterGemma.initialize(
  inferenceEngines: [OnnxEngine()],
  embeddingBackends: [OnnxEmbeddingBackend()],
);

await FlutterGemma.installEmbedder()
    .modelFromAsset('assets/models/all-MiniLM-L6-v2.onnx')
    .tokenizerFromAsset('assets/models/tokenizer.json')
    .install();

Classes

OnnxEmbeddingBackend

ONNX Runtime embedding backend — web arm (onnxruntime-web, WordPiece .onnx/.ort exports only in v1 — see onnx_web_tokenizer_loader.dart).

OnnxEngine

ONNX Runtime GenAI on-device inference engine — web arm.

OnnxHuggingFaceResolver

HuggingFaceResolver for ONNX (ORT-GenAI) models.

Typedefs

HfFetch = Future<String> Function(Uri url, Map<String, String> headers)

A GET returning the response body as text. OnnxHuggingFaceResolver accepts one so tests (and apps with their own HTTP stack, proxies, or timeout policy) can replace the default; the defaults live in hf_fetch_io.dart / hf_fetch_web.dart.

Exceptions / Errors

HfFetchException

A Hugging Face GET that failed — carries the status so the resolver can tell "repo/path does not exist" (404) apart from auth/network trouble.