flutter_edge_ai_litertlm 1.13.3

LiteRT-LM (.litertlm) engine for flutter_edge_ai: dart:ffi on 5 native platforms plus web, CPU/GPU/NPU, and the LiteRT embedding backend.

flutter_edge_ai_litertlm

Renamed from flutter_gemma_litertlm. Same package, new name: swap the dependency and the package:flutter_gemma_litertlm/ imports; nothing on the device changes. See the migration guide.

LiteRT-LM on-device engine for flutter_edge_ai: runs .litertlm models, EmbeddingGemma 2 .litertlm embeddings and LiteRT .tflite embeddings. Opt-in package — add it only if you run one of them. Android, iOS, macOS, Linux and Windows via dart:ffi; Web (early preview) via @litert-lm/core and LiteRT.js.

This package owns the shared LiteRT-LM native library (libLiteRtLm) and exposes the LiteRT C API bindings (LiteRtBindings); both are shared by flutter_edge_ai_speech. It also ships the LiteRT C API embedding backend (LiteRtEmbeddingBackend) — see Embeddings below — over the runtime-agnostic embedding pipeline in flutter_edge_ai. Tokenizers for .tflite models come from flutter_edge_ai_embeddings, which the app registers; this package does not depend on it.

Teach your AI assistant this package

dart run skills@ get --all

Installs the agent skills flutter_edge_ai bundles — this package depends on it, so they come with it. One of them, flutter-edge-ai-inference, covers the .litertlm engine, installing a model from Hugging Face, sessions, streaming, and the platform setup for all six targets.

Usage

import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';

await FlutterEdgeAi.initialize(
  inferenceEngines: [LiteRtLmEngine()],
);

LiteRtLmEngine handles ModelFileType.litertlm models; pass it alongside other engines (e.g. MediaPipeEngine from flutter_edge_ai_mediapipe) if your app uses both formats.

Depend on both packages (flutter_edge_ai: ^2.3.2, flutter_edge_ai_litertlm: ^1.13.2) and install .litertlm files with fileType: ModelFileType.litertlm: installModel defaults to ModelFileType.task, which routes the file to MediaPipe.

Since 1.13.0 this package needs flutter_edge_ai 2.3.0 (^2.3.1 since 1.13.1), and 2.3.0 needs every flutter_edge_ai package upgraded together: flutter_edge_ai_embeddings 2.3.0 · flutter_edge_ai_onnx 0.5.8 · flutter_edge_ai_mediapipe 1.2.2 · flutter_edge_ai_builtin_ai 0.3.5 · flutter_edge_ai_speech 0.5.6 · flutter_edge_ai_agent 0.2.9 · flutter_edge_ai_rag 1.1.0 · flutter_edge_ai_sqlite 2.1.0 · flutter_edge_ai_qdrant 2.1.0 · genkit_flutter_edge_ai 0.9.1. Run flutter pub upgrade, not flutter pub upgrade flutter_edge_ai_litertlm. The migration guide lists the API changes.

Sampling

Leave temperature, topK, topP and randomSeed unset on createSession or createChat and each comes from the sampler the .litertlm file ships, else from the model family's defaults (SamplingParams.forModelType), else temperature: 0.8, topK: 40, topP: 0.95, randomSeed: 1; a value you pass wins, field by field. topK: 1 or temperature: 0 decodes greedily. Before core 2.2.0 an unset topK was 1, so every session decoded greedily.

• LiteRT-LM keeps the sampler of an engine's first generation, random state included, for every later conversation (LiteRT-LM #2080): set sampling before the first generation, or close and reload the model to change it. A warning says so once per loaded model when a later session asks for other values and either set was yours.
• On web the file's sampler is not read, so unset values come from the family defaults. The values you pass are applied; before 1.12.0 web dropped them.
• On the NPU decoding is greedy whatever you pass; a one-time warning says so when you set any value.

Context and reply length

maxTokens on getActiveModel is the whole context window — prompt, history and reply — not the reply length. On CPU and GPU a value below 1024 is raised to 1024, because the published bundles fail to allocate below it; on the NPU it is passed through, so use the context size the bundle was compiled for. Cap the reply with createSession(maxOutputTokens:).

Install from a Hugging Face repo (litertlm_manifest.json)

Repos that ship a litertlm_manifest.json deployment manifest describe every .litertlm file they contain — which backends each is verified on, which file a given platform should pick, sha256/ size identity, and session guidance. LitertlmManifestResolver reads it so an app installs "the right file for this device" without hardcoding filenames:

import 'dart:math' show max;

// LiteRtLmEngine carries this resolver, so registering the engine registers
// it too. Pass huggingFaceResolvers: only to override — e.g.
// [LitertlmManifestResolver(revision: 'abc123')] to pin a commit.
await FlutterEdgeAi.initialize(inferenceEngines: [LiteRtLmEngine()]);

final r = await FlutterEdgeAi.resolveHuggingFace(
    'litert-community/Qwen3-4B-Thinking-2507',
    fileType: ModelFileType.litertlm);
await FlutterEdgeAi.installModel(
      // The manifest types 2507 as qwen3; ModelType.qwen is right for it.
      // For other repos, r.modelType ?? ModelType.general.
      modelType: ModelType.qwen,
      fileType: r.fileType,
    )
    .fromNetwork(r.url) // authoritative: carries the resolver's revision pin
    .install();
final model = await FlutterEdgeAi.getActiveModel(defaults: r.runtime);
final session = await model.createSession(
  enableThinking: r.runtime.thinkingDeclared ?? false,
  // minOutputTokens is a floor, not a cap: keep the app's own budget unless
  // the manifest asks for more.
  maxOutputTokens: max(1024, r.runtime.minOutputTokens ?? 0),
);

Everything the manifest returns is an overridable default (explicit argument > manifest > SDK default); r.notes carries platform caveats and known issues worth surfacing to developers. Repos without a manifest keep working through installModel(...).fromHuggingFace(repo, file: ...).

To resolve and install in one step, omit file: fromHuggingFace(repo) reads the manifest at install time, installs the revision-pinned variant, and returns the same defaults on InferenceInstallation.runtime (plus notes). The two-step form above stays the offline-safe one — manifest mode needs the network on every install, because the variant's filename is only known after the fetch.

Embeddings

LiteRtEmbeddingBackend runs two kinds of embedding model:

Model file	Platforms	Tokenizer	Longest input
EmbeddingGemma 2 text, images and audio, one .litertlm	Android, iOS, macOS, Linux, Windows; Web via @litert-lm/core (WebGPU); per-SoC bundles on Qualcomm NPUs (Android, Linux arm64) and Intel Panther Lake NPUs (Windows x64)	inside the bundle	1024 tokens
EmbeddingGemma 1, Gecko: .tflite + a tokenizer file	Android, iOS, macOS, Linux, Windows; Web via LiteRT.js	flutter_edge_ai_embeddings	the model's sequence length (256 for a seq256 file)

import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';

await FlutterEdgeAi.initialize(
  embeddingBackends: [LiteRtEmbeddingBackend()],
  embeddingTokenizers: [GemmaEmbeddingTokenizers()], // .tflite models only
);

final embedder = await FlutterEdgeAi.getActiveEmbedder();
final vector = await embedder.embed(
  EmbeddingInput.text('How do I reset the device?'),
  taskType: TaskType.retrievalQuery,
);

A .tflite model needs both lists and both packages: this one brings the backend, and flutter_edge_ai_embeddings brings the tokenizers it asks core for. On web the tokenizer list is unused — the LiteRT.js bundle tokenizes in JS. The pipeline the backend plugs into — the embedding pass SPI, the host that runs a pass off the UI isolate, and the pooling — lives in flutter_edge_ai. On web it runs via LiteRT.js instead; see Embeddings on web below for the four files and the <script> tag your app needs.

taskType is required and prefixes the text: retrievalQuery and retrievalDocument for search, none for no prefix, and questionAnswering, factChecking, codeRetrieval, classification, clustering, sentenceSimilarity. Every model here gets the same prefix for the same task, native and web.

An input longer than embedder.maxInputTokens — task prefix, BOS and EOS included — is never truncated: embed fails with EmbeddingInputTooLongException. Every embedding error arrives as a failed future (EmbeddingException and its subclasses).

EmbeddingGemma 2

EmbeddingGemma 2 ships as one .litertlm bundle that carries its own tokenizer, so it installs without tokenizerFrom…() and needs neither flutter_edge_ai_embeddings nor embeddingTokenizers:. The text-only 270m bundle is the one for text; the 440m adds images (below) and the 740m adds audio (below). The repositories are not gated:

const revision = '9be6e8b90982095dc05c2bd162e4b954ee4dbac7';

await FlutterEdgeAi.initialize(
  embeddingBackends: const [LiteRtEmbeddingBackend()],
);
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/'
      'embeddinggemma-2-text-270m-litert-lm/resolve/$revision/'
      'embeddinggemma-2-text-270m.litertlm',
    )
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder();
final vector = await embedder.embed(
  EmbeddingInput.text(chunk), // core adds the task prefix
  taskType: TaskType.retrievalDocument,
);

• Android, iOS, macOS, Linux and Windows, and the web (1.13.1). On the web the bundle runs through @litert-lm/core's EmbeddingEngine on WebGPU, which needs one more line in web/index.html — see Web setup — and a browser with WebGPU.
• Text on every bundle; images on the 440m and the 740m; audio on the 740m. Natively all from 1.13.1; on the web text and images from 1.13.1, audio and combined inputs from 1.13.2. On the 270m, getActiveEmbedder(supportImage: true) fails with EmbeddingLoadException naming the 440m bundle, and on the 270m and the 440m supportAudio: true fails naming the 740m — read from the bundle's own header before any engine or cache is made, on the web too (a few KB of the installed file).
• At most 1024 tokens per input, task prefix, BOS and EOS included (maxInputTokens == 1024). Keep chunks at 512 tokens or fewer: memory grows with the input length (on an M4 Pro Mac about +170 MB at 128 tokens, +340 MB at 512 and +560 MB at 1024). A longer input fails with EmbeddingInputTooLongException whose actualTokens is the exact count. Natively (from 1.13.1) the engine reports it from a truncated run; on the web it reports it after running the whole input in 1024-token windows (WebGPU averages them instead of truncating; the averaged vector is never returned). Either way an over-long input costs a full embedding before it is refused. Split before embedding.
• CPU natively, WebGPU on the web. activeBackend is PreferredBackend.cpu on Android, iOS and desktop and PreferredBackend.gpu on the web; preferredBackend is not applied — except npu, which this bundle refuses: the NPU runs the per-SoC bundles (below). The web has no CPU path for these bundles: the CPU (WebAssembly) backend of @litert-lm/core 0.18.0 cannot read their external weights, so a browser without WebGPU (no navigator.gpu, or no GPU adapter) fails getActiveEmbedder() with an EmbeddingLoadException that says so, before any engine is created. Web vectors come from WebGPU and differ slightly from the native CPU ones: measured min cos about 0.995 for text and 0.998 for images (Chrome 155, Tesla T4). Retrieval across platforms holds: web text queries find natively embedded images and native queries find web-embedded images, top-1 10/10 on the example's ten scenes.
• Disk: the 165 MB bundle plus a 71 MB weight cache the engine writes at its first load. The 440m bundle adds a 218 MB vision cache and the 740m bundle writes 385 MB of caches in total, even for text-only use. The cache lives in the app's cache directory: it is never backed up, and if the OS clears it the next load writes it again. On the web there is no weight cache: the bundle sits in the browser's storage and the engine rebuilds its weights at every load.
• EmbeddingGemma 1 and 2 both produce 768-dimensional vectors in different embedding spaces: never mix them in one vector store.

EmbeddingGemma 2 images

The 440m bundle carries the 270m's text tower (the same text vectors) plus a vision encoder, so one model embeds text and images and a text can find a photo. Declare images when you get the embedder:

const visionRevision = 'e301f74d5551b0c2641bd5cb4652a76239d5c5f8';

await FlutterEdgeAi.initialize(
  embeddingBackends: const [LiteRtEmbeddingBackend()],
);
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/'
      'embeddinggemma-2-text-vision-440m-litert-lm/resolve/$visionRevision/'
      'embeddinggemma-2-text-vision-440m.litertlm',
    )
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder(supportImage: true);
final photo = await embedder.embed(
  EmbeddingInput.image(await uprightAndSmall(photoBytes)), // below
  taskType: TaskType.retrievalDocument, // prefixes reach text only
);
final query = await embedder.embed(
  EmbeddingInput.text('a red bicycle'),
  taskType: TaskType.none, // a text query over images works best raw
);

Pass upright, downscaled images. LiteRT-LM decodes the file itself and ignores EXIF orientation, so a portrait photo from a phone camera — stored sideways with an orientation tag — is embedded lying on its side (measured: cos 0.89 against the same picture upright). It also decodes at full size, which a 12 MP photo pays for in time and memory the model does not need: it scales every image down to its vision budget anyway. Flutter's codec fixes both:

import 'dart:math' as math;
import 'dart:typed_data';
import 'dart:ui' as ui;

/// [photo] upright (Flutter applies the EXIF orientation) and at most
/// 1024 px on its long side, as PNG.
Future<Uint8List> uprightAndSmall(Uint8List photo) async {
  final codec = await ui.instantiateImageCodecWithSize(
    await ui.ImmutableBuffer.fromUint8List(photo),
    getTargetSize: (w, h) => math.max(w, h) <= 1024
        ? const ui.TargetImageSize()
        : w >= h
        ? const ui.TargetImageSize(width: 1024)
        : const ui.TargetImageSize(height: 1024),
  );
  final image = (await codec.getNextFrame()).image;
  final png = await image.toByteData(format: ui.ImageByteFormat.png);
  image.dispose();
  codec.dispose();
  return png!.buffer.asUint8List();
}

It also converts what Flutter decodes and the engine does not (WebP; HEIC on Android). The example app writes BMP instead of PNG — cheaper to write and to read back — and gets the camera photo's vector within cos 0.998 of the upright one. The same code runs on the web (the browser decodes, through CanvasKit or skwasm); there are no isolates there, so the example writes its BMP on the page.

• Formats the engine decodes: JPEG, PNG, GIF and BMP. HEIC, WebP, TIFF and AVIF fail with EmbeddingFormatNotSupportedException (which names the detected and the accepted formats) before anything is sent. A corrupt or truncated file of an accepted type fails with EmbeddingEngineException (nativeCode 3 natively; on the web the message carries the runtime's reason) and the embedder keeps working. One image per vector, unless you send several as one combined input (natively, from 1.13.1; below).
• Vision budget: 140 tokens per image by default, the closest to the reference model (min cos 0.985 on our test images). LiteRtEmbeddingBackend(visionTokensPerImage: 70) — what the model card and Google AI Edge Gallery use — is about twice as fast with slightly lower fidelity (min cos 0.970). Any other value fails with EmbeddingLoadException. Text vectors do not depend on it. Image vectors do, so give an image store an id that names the budget: the text profile id plus -vis140 (or -vis70).
• Next to an LLM on the web the vision encoder does not fit: see Web setup. Close the .litertlm LLM before loading the 440m with images, or embed text only.
• Cost (CPU, measured): a 388 MB bundle; at every load the engine writes a 218 MB vision weight cache next to the 71 MB text cache in the app's cache directory, also when you only embed text. The first image adds about 385 MB of RAM on an M4 Pro Mac and about 500 MB on a Pixel 8. Per image at 140 tokens: about 0.35 s on an Apple-silicon Mac, about 2 s on an Android flagship (Pixel 8 Pro 1.8 s, Pixel 8a 2.3 s), more on mid-range phones and rising under sustained load; 70 tokens takes about half. On the web (WebGPU, Tesla T4) an image takes about 0.33 s and a short text about 0.12 s. LiteRT-LM's thread setting reaches the text tower only; images run with its default threads. A 440m loaded for text alone still loads its vision tower: for text use the 270m.
• The example app's Image Search screen indexes photos (or ten bundled sample pictures) in a RAG image index and searches them by text and by photo; choose "EmbeddingGemma 2 Text + Vision (440M)" under Embedding Models first.

EmbeddingGemma 2 audio and combined inputs

The 740m bundle carries the 440m's text and vision towers (the same text and image vectors) plus an audio encoder, so a text can find a spoken sentence or a sound, and one vector can hold several parts — a photo with its caption, a clip with a note. Natively from 1.13.1, on the web (WebGPU) from 1.13.2. Declare what you send when you get the embedder:

const audioRevision = '24d962e906c7d332c6428e71c9676855024569e2';

await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/'
      'embeddinggemma-2-740m-litert-lm/resolve/$audioRevision/'
      'embeddinggemma-2-740m.litertlm',
    )
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder(supportAudio: true);
final clip = await embedder.embed(
  EmbeddingInput.audio(wavBytes),
  taskType: TaskType.retrievalDocument, // prefixes reach text only
);
final query = await embedder.embed(
  EmbeddingInput.text('turn off the kitchen lights'),
  taskType: TaskType.none, // a text query over media, as rag sends it
);

// Several parts, one vector (flutter_edge_ai 2.3.1):
final note = await embedder.embed(
  EmbeddingInput(text: 'kitchen', audio: wavBytes),
  taskType: TaskType.retrievalDocument,
);

• Formats the engine decodes: WAV, MP3 and FLAC, any sample rate and channel count — the engine mixes down to mono and resamples to 16 kHz. Ogg and M4A fail with EmbeddingFormatNotSupportedException before anything is sent; a corrupt file of an accepted type fails with EmbeddingEngineException (natively nativeCode 2; on the web it carries the runtime's message, "Failed to initialize miniaudio decoder") and the embedder keeps working. The web decodes with the same miniaudio, compiled to WebAssembly. One clip per vector.
• Length: about 25 tokens per second of audio, so a clip of up to about 40 seconds fits the 1024-token cap; a 60-second clip fails with EmbeddingInputTooLongException(maxInputTokens: 1024, actualTokens: 1504), natively and on the web alike. Split long recordings.
• Combined inputs: EmbeddingInput(text:, images:, audio:) puts the text (with its task prefix), then the images in order, then the audio into one vector — that order whatever order you write the arguments in. Several images in one input are a combined input too. Declare every medium you send (supportImage: true, supportAudio: true); the cap counts all parts together (about 140 tokens per image), on the web too.
• Cost (CPU, measured on an M4 Pro Mac): a 485 MB bundle; at every load the engine writes 385 MB of weight caches (text 71 MB, vision 218 MB, audio 95 MB) in the app's cache directory, also when you only embed text; about 0.2–0.3 s per 3-second clip. On the web (WebGPU, Tesla T4, Chrome 155) getActiveEmbedder(supportAudio: true) on the 740m took about 17 s, of which the engine's create is about 5 s (9 s with images too; no weight cache there); a 3-second clip takes about 0.3 s, the first one about 4 s while WebGPU compiles. The audio tower runs on WebGPU: on the CPU (WebAssembly) it takes about 18 s per clip, on the page's main thread. Not measured next to a .litertlm LLM on the same page. For text use the 270m, for text and images the 440m.
• Index ids: audio vectors do not depend on the vision budget, so an audio store keeps the text profile id (rag tells the stores apart by their documents modality).
• Audio vectors are not bit-portable. The audio encoder quantizes its activations to 8 bits, so a difference in the last bit of a float — another CPU kernel, or one sample of the clip changed by 1 — can move a clip's vector by up to cos 0.05. Text and image vectors do not move that way. Our native build and Google's litert-lm Python wheel give identical text vectors and audio vectors within cos 0.97 (the wheel even varies by thread scheduling run to run); one build is deterministic on one platform. The web (WebGPU) is within cos 0.97 of both. What holds across builds is retrieval: a text finds its clip on every build we measured, the web included, and a web query finds the clips embedded natively.

EmbeddingModel.activeBackend is cpu on native and gpu for EmbeddingGemma 2 on the web, and npu for an EmbeddingGemma 2 per-SoC bundle on a Qualcomm NPU (below) or an Intel one (below); any other preferredBackend is not applied. For EmbeddingGemma 1 on the web it is null and LiteRT.js picks: window.getLiteRtEmbeddingAccelerator() names where the output buffer lived after the first embedding, and window.getLiteRtEmbeddingFullyAccelerated() says whether the graph landed entirely on the requested accelerator — false also when LiteRT silently recompiled a WebGPU request for WASM.

EmbeddingGemma 2 on a Qualcomm NPU

Google publishes every EmbeddingGemma 2 bundle a second time, compiled for one SoC's NPU, in the same repositories: embeddinggemma-2-text-270m_Qualcomm_SM8850.litertlm, embeddinggemma-2-text-vision-440m_Qualcomm_QCS8275.litertlm, and so on. Such a bundle runs only on that SoC's NPU. This package runs it on Qualcomm Android (arm64) and Qualcomm Linux arm64:

const revision = '9be6e8b90982095dc05c2bd162e4b954ee4dbac7';
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/'
      'embeddinggemma-2-text-270m-litert-lm/resolve/$revision/'
      'embeddinggemma-2-text-270m_Qualcomm_SM8850.litertlm',
    )
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder(
  preferredBackend: PreferredBackend.npu,
);
assert(embedder.activeBackend == PreferredBackend.npu);

• Opt in to the Qualcomm stack, as for the LLM NPU path: qualcomm_npu: true under hooks: user_defines: flutter_edge_ai_litertlm: (Qualcomm NPU).
• Pick the bundle for the device's SoC. Verified on device: SM8850 (Snapdragon 8 Elite Gen 5), SM8650 (Snapdragon 8 Gen 3) and QCS8275 (Arduino VENTUNO Q). Google also publishes SM8550, SM8750, Tensor G5/G6, MediaTek MT6991/MT6993 and Intel PTL builds; the Qualcomm ones load here but are not verified by us, Intel PTL runs on Windows (below), and the others are refused (no dispatch for them ships). What a bundle is compiled for is read from the bundle itself, not from its name.
• Fast, and close to — not equal to — the CPU vectors. Measured on an SM8650 (Galaxy S24): a text in about 9 ms (53 ms on its CPU), an image in about 0.31 s, the 270m loaded in 0.85 s. Against the CPU bundle, the cosine is 0.79–0.86 for sentences of a few words, 0.92 at about 50 words and 0.99 from about 200; images 0.86–0.88. Retrieval within the NPU's own space holds: ten of ten captions find their picture, every query its document.
• Nothing switches backend. NPU and CPU vectors are not interchangeable, so every mismatch is an EmbeddingLoadException that says what to do: a per-SoC bundle without PreferredBackend.npu; the generic bundle with npu; npu on iOS, macOS, Linux x64 or the web; a Qualcomm bundle on Windows; an app built without qualcomm_npu; a device without a reachable Qualcomm NPU; a bundle the NPU cannot run (another SoC's — the load embeds one test text to find out). A loaded embedder is never handed to a request for the other backend.
• Give an NPU index its own profile id. Vectors from the SM8850 bundle and from the generic bundle live in different spaces, so a RAG store built with one never takes the other's queries. The example app uses the CPU text id plus -npu-<soc> (embeddinggemma2-litertlm-text-4410a540ce75-prefix-v1-npu-sm8850, and …-vis140 for an image index); the 270m and the 440m of one SoC share it, because their text towers are byte-identical. FlutterEdgeAiActiveEmbedder.forActiveModel keys on the installed file name, so it separates them by itself.
• Text up to 1024 tokens (the bundles carry no longer signature), and images on the 440m, as on CPU; audio on the 740m's per-SoC bundles from 1.13.2, with their audio encoder on the CPU (verified on SM8850, Galaxy S26+).
• On Linux the NPU calls run with SIGPROF blocked, as the LLM engine's create does: QNN does not survive the debug VM's profiler signal. Linux needs native-v0.18.0-e or newer, which reads the bundles' external weights by path.

EmbeddingGemma 2 on an Intel NPU (Windows)

Google's …_Intel_PTL.litertlm bundles are compiled for the Intel Panther Lake NPU. From 1.13.2 this package runs them on Windows x64 with PreferredBackend.npu, through the Intel stack (LiteRtDispatch.dll, OpenVINO and TBB) that every Windows x64 build already carries beside the executable — no opt-in flag:

const revision = '9be6e8b90982095dc05c2bd162e4b954ee4dbac7';
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/'
      'embeddinggemma-2-text-270m-litert-lm/resolve/$revision/'
      'embeddinggemma-2-text-270m_Intel_PTL.litertlm',
    )
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder(
  preferredBackend: PreferredBackend.npu,
);

• The load checks the NPU. It embeds one test text and refuses unless it gets a valid vector, so a PC without an Intel NPU, or one the stack cannot drive, gets an EmbeddingLoadException rather than wrong vectors.
• The same rules as on Qualcomm: refused without npu, refused on any other platform (an Intel bundle on Android or Linux, a Qualcomm bundle on Windows), refused with the reason when LiteRtDispatch.dll is not beside the executable, and never moved to CPU.
• Its own profile id. The example app uses the CPU text id plus -npu-ptl (and …-vis140 for an image index); the 270m and the 440m share it, because their text towers are byte-identical.

Embeddings on web

EmbeddingGemma 2 (.litertlm) on the web needs no files from this package — only the window.EmbeddingEngine line in Web setup. The rest of this section is about EmbeddingGemma 1 and Gecko (.tflite).

On web, flutter_edge_ai_litertlm's .tflite embedding backend runs via LiteRT.js. Copy all four files from this package's web/ into your app's web/, next to index.html — litert_embeddings.js imports the other three by relative path, so they have to sit together:

litert_embeddings.js  sentencepiece.js  litert.js  tensorflow.js

They are four pieces of one bundle (the entry plus three vendor chunks), built together by tool/web_build, so never mix them across package versions. Find this package's directory with grep -A1 '"name": "flutter_edge_ai_litertlm"' .dart_tool/package_config.json, then load the entry module from web/index.html:

<script type="module" src="litert_embeddings.js"></script>

Upgrading from an earlier version: delete the copies in your app's web/ and re-copy all four from this package. 1.13.0 changed the bundle's API: Dart now builds the task prefix, so every TaskType gets the prefix and token ids it gets on native, and the bundle reports the model's input limit instead of truncating. A copy from an earlier version fails getActiveEmbedder() with an EmbeddingLoadException that names litert_embeddings.js. Before flutter_gemma_litertlm 1.8.0 they came from flutter_gemma_embeddings, and the copies you have are built against an older @litertjs/core than the runtime this version loads. If you built your own web/wasm/, either delete it and take the CDN default or rebuild it from the version in LiteRtWebRuntime.pinnedVersion.

Loading litert_embeddings.js straight from a CDN with a Subresource-Integrity hash — which an older README suggested — cannot work: the module's three imports resolve against the CDN path, where they do not exist, so the module never executes and every embedding call fails on an undefined global. SRI would not have covered the imports either.

The WASM runtime

LiteRT.js loads a WASM runtime at the first embedding call — litert_wasm_internal.js, or litert_wasm_compat_internal.js on a browser without relaxed SIMD, each with a ~9 MB .wasm beside it. Since flutter_gemma_litertlm 1.8.0 they come from the pinned @litertjs/core build on jsDelivr by default — nothing to install, and nothing this package has to carry into every native-only app.

To serve them yourself (offline, an air-gapped deploy, or a CSP that forbids third-party script), copy node_modules/@litertjs/core/wasm/ into your app's web/wasm/ and point the package at it before the first embedding:

import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';

LiteRtWebRuntime.wasmPath = '/wasm/';

Those files come from @litertjs/core — npm i @litertjs/core@2.5.3 in a scratch directory, then copy its wasm/.

Set the prefix before the first embedding — the runtime is loaded once and cached, so a later assignment is ignored. LiteRT.js inserts the separator when it joins the prefix with the file name, so the trailing slash above is convention, not a requirement; the value is root-absolute, and an app served under a base href other than / needs /my-app/wasm/ or a full URL.

Pin @litertjs/core to LiteRtWebRuntime.pinnedVersion if you vendor it. The runtime and this package's web/litert.js are two halves of one release — litert.js calls that release's WASM entry points by name — and a mismatch fails at the first embedding with something that does not mention versions at all: a runtime older than the glue gives Cannot read properties of undefined (reading 'create').

Serving it yourself is also the answer if a third-party script in your app's runtime path is not acceptable to you: LiteRT.js injects the <script> itself, so the CDN copy carries no Subresource-Integrity hash.

Whatever host you use must send Access-Control-Allow-Origin (LiteRT.js sets crossOrigin="anonymous" on the script it injects) and serve .wasm as application/wasm.

Native embeddings need no web files; the platform setup under Platforms still applies.

Web setup (early preview)

.litertlm web inference runs via @litert-lm/core (WebGPU/WASM). Load a model's web export (gemma-4-E2B-it-web.litertlm, not gemma-4-E2B-it.litertlm). It is text-only: an image or audio message throws UnsupportedError, because the JS API creates the LLM engine with no vision or audio executor, and loraPath throws too. createSession(maxOutputTokens:) is honoured here as it is on native. Add the handshake below to your app's web/index.html <head> — the ESM doesn't assign window globals and module scripts are deferred, so Dart awaits window.litertLmReady (which resolves to the Engine constructor):

<script type="module">
window.litertLmReady = (async () => {
  const m = await import('https://cdn.jsdelivr.net/npm/@litert-lm/core@0.18.0/+esm');
  window.Engine = m.Engine;
  window.EmbeddingEngine = m.EmbeddingEngine;
  return m.Engine;
})();
</script>

window.EmbeddingEngine (1.13.1) is what EmbeddingGemma 2 .litertlm embedders run on, on WebGPU: text and images, and from 1.13.2 audio and combined inputs (no further line). A page without that line fails the embedder's load with an EmbeddingLoadException that names it; so does a browser without WebGPU, which has no fallback here (the CPU path of @litert-lm/core 0.18.0 cannot read these bundles' external weights). The LLM engine does not need the line.

An LLM and an EmbeddingGemma 2 embedder on the same page share one WebAssembly module and one WebGPU device, so they take turns and their GPU memory adds up: an embedding, or an embedder's load, waits while a reply is being generated (until its stream ends, even if you stopped listening), and a reply waits for an embedding in progress. The module has 4 GB of memory (WebAssembly is 32-bit), and the 440m's vision encoder does not fit in it next to Gemma 4 E2B (measured: the vision engine alone takes about 3.3 GB): whichever loads second fails, the embedder with an EmbeddingLoadException that says to close the LLM first. A text-only embedder — the 270m, or the 440m without supportImage — loads next to the LLM.

Native platforms need no web setup.

Platforms

Platform	Support
Android	✅ FFI (GPU via OpenCL, NPU on Qualcomm — opt-in, below)
iOS	✅ FFI (GPU via Metal on device; CPU on simulator)
macOS / Linux	✅ FFI (GPU via Metal / Vulkan; NPU on Qualcomm Linux arm64 — opt-in, below)
Windows	✅ FFI on x64 (CPU + GPU via DirectX 12 + Intel NPU); ARM64 has no native archive, see Troubleshooting
Web	⚠️ early preview via @litert-lm/core (CDN; LLMs text-only; EmbeddingGemma 2 text, images and audio on WebGPU)

The native library (native-v0.18.0-e, LiteRT-LM v0.18.0) is fetched at build time by hook/build.dart from a SHA256-verified GitHub release. The app still needs its platform setup: Android minSdk 30, iOS 15.0 and the memory entitlements, and on macOS a post_install block in macos/Podfile, without which the first model load fails with Library not loaded: @rpath/libGemmaModelConstraintProvider.dylib. See platform setup.

PreferredBackend.npu is attempted only on Windows, on Android devices with Qualcomm FastRPC (libcdsprpc.so) and on Qualcomm Linux arm64 boards, the last two when the app opted in (below); elsewhere it falls back to GPU, then CPU, and prints why. On Windows the check is per OS, so a PC without an Intel NPU can report activeBackend == npu while the model runs elsewhere.

Run a Gemma 4 bundle on the NPU: a Gemma 3 bundle drops every prefill chunk after the first, with no error, and answers from the start of the prompt alone (LiteRT-LM#3508).

On macOS, App Nap throttles an app that is in the background: after about a minute, generation and embedding run several times slower (measured: an EmbeddingGemma 2 image went from 0.35 s to 2–7 s). An app that embeds or generates while in the background should hold an activity for the duration — ProcessInfo.processInfo.beginActivity(options: .userInitiated, reason: …) in Swift, ended with endActivity when the work is done.

Qualcomm NPU on Android and Linux arm64 (opt-in)

This package does not ship Qualcomm's QNN runtime: Qualcomm licenses it for redistribution inside an application only. To use PreferredBackend.npu on Snapdragon, opt in from your app's pubspec.yaml:

hooks:
  user_defines:
    flutter_edge_ai_litertlm:
      qualcomm_npu: true

Put it in the pubspec of the app you build. If that app is a member of a pub workspace, put it in the workspace root's pubspec instead; pub ignores user_defines anywhere else.

With the flag set, the build hook downloads com.qualcomm.qti:qnn-runtime:2.50.0 from Maven Central once, checks its SHA-256, raises the Hexagon libraries' page alignment to the 16 KB Google Play requires, and bundles the ten QNN libraries into the Android app: about 22 MB more to download and 83 MB more installed (one blob per Hexagon generation, V73 to V81). The first NPU run copies them out of the APK once more, because the DSP loads them from files, so a phone that uses the NPU holds about 166 MB of them. Setting the flag means accepting Qualcomm's AI Stack License, the LICENSE.pdf the hook prints the path of. Its notices are in this package's NOTICES, so they reach your app's licence page.

Behind a proxy, the hook honours HTTPS_PROXY. Without access to Maven Central, point it at a mirror or at the AAR itself:

hooks:
  user_defines:
    flutter_edge_ai_litertlm:
      qualcomm_npu: true
      qualcomm_npu_maven_url: https://maven.example.com/maven2
      # or: qualcomm_npu_aar: third_party/qnn-runtime-2.50.0.aar

qualcomm_npu_aar is resolved against the pubspec that declares it and must be the same file as Maven's (same SHA-256). Without the flag, a request for PreferredBackend.npu on Android falls back to GPU, then CPU, and the log says how to enable it.

Linux arm64. The same flag bundles the stack into Linux arm64 builds, for Qualcomm Linux boards with a Hexagon V68–V81 compute DSP (QCS6490, QCS8275, QCS9075, …). Qualcomm publishes no Maven artifact for Linux, so the hook reads the fourteen libraries it needs (libQnnHtp, libQnnSystem and a Stub/Skel pair per Hexagon version) out of Qualcomm's public QAIRT 2.50 SDK zip with HTTP range requests — about 32 MB of a 2.6 GB archive — checks each against a pinned SHA-256, and caches them; about 90 MB more installed. Offline, download v2.50.0.260828.zip yourself and point the hook at it:

hooks:
  user_defines:
    flutter_edge_ai_litertlm:
      qualcomm_npu: true
      qualcomm_npu_qairt_zip: third_party/v2.50.0.260828.zip

On the board, Qualcomm's FastRPC library must be installed (qcom-fastrpc1 on Ubuntu; Qualcomm's images ship it) and the user must be in group fastrpc (sudo usermod -aG fastrpc $USER, then log in again). If either is missing, npu falls back to GPU, then CPU, and the log names what to fix. Use the model compiled for your SoC, e.g. gemma-4-E2B-it_qualcomm_qcs8275.litertlm. A Linux x64 build ignores the flag.

Embeddings. The same flag runs EmbeddingGemma 2 per-SoC bundles on the NPU (EmbeddingGemma 2 on a Qualcomm NPU); Windows runs the Intel ones without it (EmbeddingGemma 2 on an Intel NPU). Unlike the LLM path, an embedder never falls back: when the NPU cannot run the bundle, getActiveEmbedder fails with EmbeddingLoadException and the reason, because CPU vectors would land in another space.

Troubleshooting

Windows ARM64: LiteRT-LM inference is not available

Symptom: a native Windows ARM64 build (a Snapdragon X laptop) succeeds, then the first .litertlm model fails with Failed to load dynamic library 'StreamProxy.dll' (error 126). The build hook now prints why while it builds.

Cause: there is no windows_arm64 LiteRT-LM archive. Upstream does not build LiteRT-LM for Windows ARM64, so there is nothing to package. The hook skips the target instead of failing, so an app that uses another engine still builds.

What works today: MediaPipe has no desktop engine and the ONNX engine is Windows x64 only (ARM64 is not released), so the option is to build the app for x64, which runs under Windows x64 emulation. See #326.

Wrong digits from a long prompt on GPU

Gemma 4 can copy numbers wrongly out of a prompt of about 2,000 tokens or more on Metal and Adreno, the same way on every run. The published Gemma 4 .litertlm files ask for half-precision activations; pass activationDataType: ActivationDataType.float32 to getActiveModel for full precision.

Qwen3-0.6B.litertlm fails to load

Symptom: since 1.7.0 the engine fails to create for Qwen3-0.6B.litertlm from litert-community/Qwen3-0.6B, with piece must not include null character in the native error.

Cause: the SentencePiece version in LiteRT-LM since v0.17.0 rejects one piece of that bundle's tokenizer (LiteRT-LM #3865). Only this file is affected; Qwen3 0.6B itself works.

Fix: install Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm from the same repository instead (measured working on native-v0.18.0-e). See #659.

Linux: any tool call aborts the app (fixed in 1.11.1)

Up to 1.11.0, a chat or session with tools aborted on its first constrained token on Linux x64 and arm64: the prebuilt constraint provider uses another C++ ABI than the runtime. 1.11.1 (native-v0.18.0-d) bridges it. See #551.

iOS/macOS: GPU crashes in apps that also use flutter_litert (fixed in 1.10.1)

In 1.10.0 the GPU backend crashed in LiteRtCreateCompiledModel when the app also used flutter_litert. 1.10.1 (native-v0.18.0-b) fixes it; no app change is needed.

Qwen keeps reasoning with thinking off, or shows <|channel>thought text (fixed in 1.9.0)

Up to 1.8.7 "thinking off" sent nothing to the template, and a Qwen3 template that reads enable_thinking treats a missing key as on: a bundle such as Qwen3-0.6B_dynamic spent its whole output budget reasoning. Bundles that declare a thought channel also streamed that reasoning into the answer as <|channel>thought…<channel|> text, and the web path sent thinking — a key no template reads. 1.9.0 sends enable_thinking both ways on every platform, and flutter_edge_ai 2.1.0 turns channel reasoning into ThinkingResponse for every ModelType and no longer appends /no_think to an audio message (Qwen3-ASR returned an empty transcript).

A stopped chat answers every later message with nothing (fixed in flutter_gemma_litertlm 1.8.1)

Symptom: after stopGeneration() in the middle of a reply — or after abandoning the response stream — every later message on that chat or session comes back empty, on Android, iOS and desktop. A new chat on the same model answers normally. (The web engine is a separate path and is not covered by this entry.)

Cause: a conversation whose generation is cancelled mid-reply stays unusable in the native runtime.

Fix: every flutter_edge_ai_litertlm release has it. The first turn after a stop runs on a fresh conversation that replays the chat's history, including whatever the stopped reply had produced. That history is replayed as text: images and audio sent in earlier turns are not, so after a stop the model can no longer see them.

Google Play rejects the app over 16 KB page sizes (fixed in flutter_gemma_litertlm 1.8.0)

Symptom: Play Console refuses the release with "Your app does not support 16 KB memory page sizes", on any app that depends on this package. Nothing fails at build or run time — the rejection happens at submission.

Cause: the Qualcomm Hexagon DSP blobs of the NPU path (libQnnHtpV{73,75,79,81}Skel.so) arrive from Qualcomm with a 4 KB p_align. Before 1.10.0 this package bundled them into every APK; Play scans lib/**/*.so and does not care that a Hexagon image is loaded by the DSP rather than mapped by the kernel.

Fix: every flutter_edge_ai_litertlm release has it. Since 1.10.0 the Skels ship only when the app sets qualcomm_npu: true, and the hook raises them to 16 KB. Check your own build with Google's check_elf_alignment.sh against the APK, not against this package.

Android GPU crashes at engine_create on Mali (fixed in flutter_gemma_litertlm 1.8.2)

Symptom: in flutter_gemma_litertlm 1.7.0–1.8.1, PreferredBackend.gpu on an Android phone with a Mali GPU (Samsung A-series, MediaTek, Google Tensor) kills the process while the model loads — SIGSEGV at pc 0 inside libLiteRtOpenClAccelerator.so. The CPU backend and Adreno GPUs are unaffected.

Cause: the OpenCL and GPU accelerators from LiteRT-LM v0.17.0 call AHardwareBuffer_allocate without declaring libandroid.so as a dependency, so Android binds the call to address 0. Only Mali takes that path.

Fix: every flutter_edge_ai_litertlm release has it; no app change is needed. See #545.

Any tool call kills the app (fixed in flutter_gemma_litertlm 1.7.1)

Symptom: in flutter_gemma_litertlm 1.7.0, a chat or session created with tools dies on the first decoded token — EXC_BAD_ACCESS / SIGSEGV inside the runtime, on every platform, CPU and GPU alike (Linux had a second cause, fixed in 1.11.1 — see above). Dart sees no exception; flutter test reports only that the test did not complete. Generation without tools is unaffected.

Cause: constrained decoding is implemented by a prebuilt companion, libGemmaModelConstraintProvider, that ships with the LiteRT-LM release. Upstream replaced the Constraint interface, and the companion published at tag v0.17.0 still implements the old one, so the runtime we build calls into the wrong vtable slot.

Fix: every flutter_edge_ai_litertlm release has it: since flutter_gemma_litertlm 1.7.1 the companion is rebuilt from upstream main.

Windows: embeddings or speech fail with status=3 (fixed in flutter_gemma_litertlm 1.7.0)

Symptom: on Windows only, LiteRtEmbeddingBackend and flutter_edge_ai_speech fail with LiteRT call failed: CreateTensorBufferFromHostMemory(...) (status=3) in flutter_gemma_litertlm 1.4.0–1.6.4. Text generation is unaffected.

Cause: LiteRT made LiteRtLayout one layout on every compiler; this package still wrote tensor shapes in the old MSVC layout on Windows.

Every flutter_edge_ai_litertlm release includes the fix. On the legacy package line, upgrade flutter_gemma_litertlm to 1.7.0 and flutter_gemma_speech to 0.5.1.

Garbled or empty streams on Android (fixed in flutter_gemma_litertlm 1.5.2)

Symptom: a generation delivers zero chunks and throws Exception: Stream error: <U+FFFD>, often followed by Callback invoked after it has been deleted and a SIGABRT that Dart cannot catch.

Cause: on Android the first dlopen of libLiteRtLm decides, for the whole process, whether its exports are reachable from the default symbol search scope, and bionic never promotes an already-loaded library afterwards. Before flutter_gemma_litertlm 1.5.2 the embeddings and speech entry point opened it locally, so an app that embedded or transcribed anything before its first generation left the stream-callback ABI probe unable to see the library — and the probe read that as "old library" and registered the wrong callback shape.

Fix: every flutter_edge_ai_litertlm release has it. If you load libLiteRtLm yourself from app or third-party code, load it before flutter_edge_ai does and with RTLD_GLOBAL. The package cannot repair that case — bionic never promotes an already-loaded library — but it no longer generates corrupt text: a .litertlm generation raises a StateError naming the condition, and embeddings or speech (which resolve through their own handle and do not need the symbols to be ambient) log a warning and carry on. See #447.

dlopen / "library not found" (libLiteRtLm)

flutter_edge_ai_litertlm is the sole owner of the shared native library (libLiteRtLm) and bundles it via its build hook — this package's own LiteRtEmbeddingBackend and flutter_edge_ai_speech both use it directly. A stale Native-Assets cache after a native version bump can leave the library unbundled, surfacing as an opaque dlopen "no such file" on the first inference. Fix with a clean rebuild:

flutter clean
rm -rf ~/Library/Caches/flutter_gemma/native        # macOS
rm -rf ~/.cache/flutter_gemma/native                # Linux
# Windows: rmdir /s /q "%LOCALAPPDATA%\flutter_gemma\native"
flutter pub get