genkit_flutter_edge_ai
Genkit Dart plugin for flutter_edge_ai — local, on-device LLM inference with any model a registered flutter_edge_ai engine runs (models), fully offline.
Renamed from
genkit_flutter_gemma. Model and embedder ids are nowflutter-edge-ai/<name>(they wereflutter-gemma/<name>);flutterEdgeAi.model(...)builds them for you. 0.7.0 drops the old Dart names;dart fix --applyrenames them. See the migration guide.
Features
- Wraps
flutter_edge_aias a Genkit model provider - Supports text generation (blocking and streaming)
- Embeddings — register a
FlutterEdgeAiEmbedderConfig, reference it withflutterEdgeAi.embedder(...) - Multimodal input — several images and one audio clip per user message, from
data:URIs,file://paths, absolute paths orhttp(s)://URLs (each download at most 20 MiB, 30 s for all of a request's downloads; a cancel aborts them). A part without acontentTypeis typed from its URI, the served Content-Type, its extension or its bytes; a part the model cannot take, media outside a user message included, is refused withINVALID_ARGUMENT, never dropped - Function calling / tool use with
toolChoicecontrol (auto,required,none) — honors Genkit's native top-leveltoolChoice - Parallel tool calls — several calls in one response, each with a
refthat no other call of the turn has, so Genkit pairs every tool response with its request; the refs stay in the history as call ids - Structured JSON output — pass an
outputSchema; the plugin writes the schema into the prompt, sends it as flutter_edge_ai'sresponseFormatwhen the loaded model enforces it, and Genkit parsesresponse.output - Context-window trimmer middleware (
trimContext) — drops oldest turns to fit the on-device KV budget - On-device routing decisions —
DeciderRouteClassifier.load(ordeciderRouteClassifierover a decider you keep) turns a flutter_edge_aiDeciderinto agenkit_hybridrouter classifier, andflutterEdgeAi.warmUploads the model before the first request - Thinking mode on models that support it
latencyMs, and token usage inresponse.usagewhere the engine counts tokens, with time to first token and tokens/s underusage.custom. A single-turn model's chat (FunctionGemma) opens a fresh session after each answer and reports none- Configurable via
@Schema()-annotated options, validated before a model loads - Errors a router can act on: a bad request is
INVALID_ARGUMENT; no model installed isUNAVAILABLE; tools or media on a model configured without them, or an engine that cannot do this here, areUNIMPLEMENTED— agenkit_hybridrouter falls back on both; an installed model the app cannot run (no engine registered for it) isFAILED_PRECONDITION
Supported Model Architectures
Any model a registered engine runs; pass its ModelType in
FlutterEdgeAiModelConfig. Which ModelType each model takes, and what it
supports: models.
Setup
genkit_flutter_edge_ai depends only on the core flutter_edge_ai package — it
stays engine-agnostic. Since the 1.0.0 architecture split (released under the
old flutter_gemma name), the inference engines and
embedding backends ship as separate, opt-in packages, and the core
registers none of them by default. Your app must add the packages it needs and
register their providers in await FlutterEdgeAi.initialize().
| Package | Provider | Add it when you use… |
|---|---|---|
flutter_edge_ai_litertlm |
LiteRtLmEngine(), LiteRtEmbeddingBackend() |
.litertlm models (every platform, incl. desktop) and LiteRT embeddings (EmbeddingGemma 1 and 2, Gecko) |
flutter_edge_ai_mediapipe |
MediaPipeEngine() |
.task / .bin models (Android, iOS, web; MediaPipe is maintenance-only upstream) |
flutter_edge_ai_onnx |
OnnxEngine(), OnnxEmbeddingBackend() |
ONNX models and embeddings |
flutter_edge_ai_llama |
LlamaEngineProvider() |
.gguf models through llama.cpp |
flutter_edge_ai_builtin_ai |
BuiltInAiEngine() |
the OS's own model (Gemini Nano, Apple Foundation Models, Phi Silica) or the browser's Prompt API (Chrome, Edge) |
flutter_edge_ai_embeddings |
GemmaEmbeddingTokenizers() |
tokenizers for .tflite and .onnx text embeddings (on the web, .onnx only; an EmbeddingGemma 2 .litertlm needs none) |
# pubspec.yaml (your app)
dependencies:
genkit: ^1.0.0
genkit_flutter_edge_ai: ^0.12.0
flutter_edge_ai: ^2.6.0
flutter_edge_ai_litertlm: ^1.20.0 # only the engines/backends you actually use
flutter_edge_ai_embeddings: ^2.3.2 # the tokenizers an embedding backend needs
flutter_edge_ai_mediapipe: ^1.3.0
// main() — register the providers from the packages you added above.
await FlutterEdgeAi.initialize(
inferenceEngines: const [LiteRtLmEngine(), MediaPipeEngine()],
embeddingBackends: const [LiteRtEmbeddingBackend()],
embeddingTokenizers: const [GemmaEmbeddingTokenizers()],
);
If you skip registration,
getActiveModelthrows aStateErrornaming the model'sModelFileTypeand the engines that are registered. Through Genkit that error comes back in the result ofai.generate, not as an exception — see the check in the Quick Start.
Quick Start
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
// Engines/backends are opt-in (see Setup) — register the ones you need.
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';
import 'package:flutter_edge_ai_mediapipe/flutter_edge_ai_mediapipe.dart';
import 'package:genkit/genkit.dart';
import 'package:genkit_flutter_edge_ai/genkit_flutter_edge_ai.dart';
// Initialize and install model (host app responsibility)
await FlutterEdgeAi.initialize(
inferenceEngines: const [LiteRtLmEngine(), MediaPipeEngine()],
embeddingBackends: const [LiteRtEmbeddingBackend()],
embeddingTokenizers: const [GemmaEmbeddingTokenizers()],
);
await FlutterEdgeAi.installModel(
modelType: ModelType.gemma4,
fileType: ModelFileType.litertlm, // the default, .task, routes to MediaPipe
).fromNetwork(
// On web, install gemma-4-E2B-it-web.litertlm from the same repository.
'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it.litertlm',
).install();
// Create Genkit with plugin
final ai = Genkit(plugins: [
GenkitFlutterEdgeAiPlugin(
models: [
FlutterEdgeAiModelConfig(
name: 'gemma-4-e2b',
modelType: ModelType.gemma4,
),
],
embedders: [
FlutterEdgeAiEmbedderConfig(name: 'embedding-gemma-300m'),
],
),
]);
// Generate
final response = await ai.generate(
model: flutterEdgeAi.model('gemma-4-e2b'),
prompt: 'Hello!',
);
final reason = response.finishReason;
if (reason != FinishReason.stop &&
reason != FinishReason.length &&
reason != FinishReason.unknown) {
throw response.cause ??
StateError(response.finishMessage ?? 'The model stopped: $reason');
}
print(response.text);
In genkit 1.0 ai.generate does not throw when the generation fails. It returns
a result whose finishReason is failed (or aborted for a cancel), with the
error in error and the original exception in cause. A missing engine, an
invalid option and a cancel all arrive this way, so check the result before you
show its text: without the check a failure prints an empty line.
Configuration
Pass FlutterEdgeAiModelOptions to customize inference:
final response = await ai.generate(
model: flutterEdgeAi.model('gemma-4-e2b'),
prompt: 'Hello!',
config: FlutterEdgeAiModelOptions(
maxTokens: 2048,
temperature: 0.5,
topK: 40,
supportImage: true,
),
);
| Option | Type | Default | Description |
|---|---|---|---|
maxTokens |
int? |
1024 | Context window (input + output), not reply length — it goes straight into getActiveModel(maxTokens:); lowering it shrinks the KV cache. Cap the reply with maxOutputTokens. |
maxOutputTokens |
int? |
null | Cap on the tokens generated for one reply (openChat(maxOutputTokens:)). .litertlm honours it; MediaPipe ignores it. |
temperature |
double? |
model's own | Sampling temperature |
topK |
int? |
model's own | Top-K sampling; 1 is greedy |
topP |
double? |
model's own | Top-P (nucleus) sampling |
supportImage |
bool? |
false | Enable multimodal image input |
supportAudio |
bool? |
false | Enable audio input (models with an audio encoder, e.g. Gemma 4) |
enableThinking |
bool? |
false | Enable thinking mode on models that support it |
randomSeed |
int? |
1 (ONNX, or a .litertlm that ships one: the model's own) |
Random seed for sampling |
toolChoice |
String? |
'auto' |
Tool calling mode: 'auto', 'required', 'none' |
systemInstruction |
String? |
null | System-level instruction (overrides system-role messages) |
maxFunctionBufferLength |
int? |
null | Max token buffer for streaming tool-call arguments (increase for large payloads) |
enableSpeculativeDecoding |
bool? |
null | MTP speculative decoding for Gemma 4 E2B/E4B (null = model default, true/false = force on/off) |
preferredBackend |
String? |
null | Text-decoder backend: 'cpu', 'gpu', 'npu' (null = engine default) |
preferredVisionBackend |
String? |
null | Vision-encoder backend: 'cpu', 'gpu', 'npu' (null follows the bundle: its backend_constraint, else the NPU it is compiled for, else CPU; set one only where the section allows it; ignored by MediaPipe) |
preferredAudioBackend |
String? |
null | Audio-encoder backend: 'cpu', 'gpu', 'npu' (null follows the bundle, as for vision; the published Gemma 4 and Gemma 3n audio encoders declare 'cpu' and refuse 'gpu'; ignored by MediaPipe) |
Every option is checked before a model loads: a value of the wrong type, a
fraction for an integer, a maxTokens, maxOutputTokens, topK or
maxFunctionBufferLength below 1, a negative temperature, a topP outside
(0, 1], a maxOutputTokens larger than maxTokens, or a toolChoice or
backend not listed above is an INVALID_ARGUMENT.
"Model's own" is the sampler a native .litertlm file ships, else the defaults
its family publishes (SamplingParams.forModelType in flutter_edge_ai). On
.litertlm the first request after a model loads fixes the sampler for that
loaded model (LiteRT-LM#2080): a later request with different values needs the
model reloaded.
Streaming
final stream = ai.generateStream(
model: flutterEdgeAi.model('gemma-4-e2b'),
prompt: 'Write a story.',
);
final reply = StringBuffer();
await for (final chunk in stream) {
reply.write(chunk.text); // update your UI with reply.toString()
}
// A failure ends the stream normally; the result says what happened.
final result = await stream.onResult; // check finishReason as in Quick Start
Tool Use
final response = await ai.generate(
model: flutterEdgeAi.model('gemma-4-e2b'),
prompt: 'What is the weather in Paris?',
tools: [weatherTool],
);
Tool refs travel as call ids both ways (0.10.0, with flutter_edge_ai
2.4.0): a tool request's ref is the call's FunctionCallResponse.id, or
call_0, call_1, … when the engine gives none (flutter_edge_ai_llama gives its own),
and never repeats within a turn; in the history the plugin sends back, a
request's ref becomes its call's id (Message.toolCall(calls:)) and a tool
response's ref its toolCallId.
Genkit's top-level toolChoice: takes precedence over the toolChoice option.
Write it as toolChoice: .none: genkit 1.0 and flutter_edge_ai both export a
ToolChoice, so in a file that imports both without a prefix,
ToolChoice.none is an ambiguous_import error.
Structured Output
The plugin advertises output: ['text', 'json']. Genkit asks for constrained
output whenever you pass an outputSchema (a schemantic type). The plugin
always writes the schema into the prompt as instructions, the way Genkit's
simulateConstrainedGeneration() middleware does (adding the middleware as
well is harmless: the schema is written once, and the middleware turns the
native path below off). Pass outputConstrained: false to leave the prompt
alone.
When the request has no tools and no enableThinking, and the loaded model
accepts the schema (InferenceModel.refusalFor returns null), the schema is
also sent to flutter_edge_ai as the chat's responseFormat: the engine
constrains decoding to it and flutter_edge_ai checks the model's reply text.
Whether the loaded model enforces a format, and which schema keywords, is its
structuredOutputSupport. A schema the model refuses, one outside
flutter_edge_ai's dialect (a type union, a recursive type, format,
oneOf/anyOf, an additionalProperties schema), or one the engine rejects at
the first send, is not an error: the request runs prompt-only, as it did
before any engine could enforce a schema. Genkit's references to a type's
definitions ($ref, $defs, and the single allOf schemantic writes for a
field with a description) are inlined first. A structured request opens a
session of its own, so it counts against the model's maxConcurrentSessions;
a prompt-only one does not.
final response = await ai.generate(
model: flutterEdgeAi.model('gemma-4-e2b'),
prompt: 'Give me a pancake recipe.',
outputSchema: Recipe.$schema, // any @Schema()-annotated type
);
final Recipe? recipe = response.output;
A structured turn that does not end in a value of the schema fails with a
GenkitException (inside ai.generate, a result with
FinishReason.failed) whose cause is flutter_edge_ai's exception, carrying
the partial or non-conforming text. None of these statuses sends the request
to another model through genkit_hybrid's router. A reply that failed its
schema would fail the same way again, and whether the prompt may go to the
cloud is the app's decision.
| flutter_edge_ai | Status |
|---|---|
IncompleteStructuredOutputException, end: outputLimit (raise maxOutputTokens) or end: contextLimit (the context window) |
OUT_OF_RANGE |
ContextLimitException: the input did not fit what the context window had left (on any request, structured or not; the chat is as before the turn) |
OUT_OF_RANGE |
… end: closed (model closed or replaced mid-turn) or end: stopped, not by your cancel |
ABORTED |
| your cancel | CancelledException |
NonConformingStructuredOutputException (an engine bug), or an engine whose structuredOutputSupport lists keywords outside the dialect |
FAILED_PRECONDITION |
Core's check covers the model's text, not response.output: Genkit's json
format reads the value with extractJson, which prefers a fenced block inside
the text and never parses the whole text first. A checked object whose string
field holds a Markdown code block printing 42 comes back as the number 42.
So validate response.output on this path too, or decode the checked text in
response.text yourself.
On the prompt-only path Genkit does not check the reply against the schema,
so validate response.output yourself. What you get depends on the reply:
- No JSON at all (or an object cut off mid-way): the result still finishes
with
FinishReason.stop,response.outputis null andresponse.errorsays why. - A JSON object with the wrong fields:
response.outputis not null andresponse.erroris null. A missing field reads as null, and reading a field of the wrong type throws. - A bare number, string or array — prose such as "Serves 4 people" is
enough, because the
4is extracted:ai.generatethrows aTypeErrorinstead of returning a result.
The schema goes into the first system message, or into the last user message
when there is none. The systemInstruction option replaces system messages;
the plugin keeps the schema and appends it to that instruction.
Context-Window Trimming
On-device models run with a fixed, small context window (maxTokens — 1024 for
most .litertlm models). A long multi-turn chat overflows it and the native
runtime fails to allocate the KV cache mid-generation. trimContext() is a
Genkit middleware that drops the oldest non-system turns before each model
call, always keeping every system message and the most recent message:
final response = await ai.generate(
model: flutterEdgeAi.model('gemma-4-e2b'),
prompt: 'Continue our conversation…',
messages: longHistory,
use: [trimContext(maxInputTokens: 800)],
);
With no arguments the budget is derived from the request's maxTokens (the
model's context window) minus the response headroom: 256 tokens, or the
request's maxOutputTokens when that is larger. A requested output schema is
held back as well, since it is written into the prompt after the trimmer runs.
Token counts are estimated with a chars / 4 heuristic; a contiguous suffix of
recent turns is kept, never a gap.
Embeddings
// Install embedding model + tokenizer (host app responsibility)
await FlutterEdgeAi.installEmbedder()
.modelFromNetwork('https://huggingface.co/.../embeddinggemma-300M.tflite')
.tokenizerFromNetwork('https://huggingface.co/.../sentencepiece.model')
.install();
// Generate embeddings
final embeddings = await ai.embed(
embedder: flutterEdgeAi.embedder('embedding-gemma-300m'),
documents: [
DocumentData(content: [TextPart(text: 'Flutter is a UI toolkit.')]),
DocumentData(content: [TextPart(text: 'Dart is a programming language.')]),
],
);
for (final embedding in embeddings) {
print('Vector (${embedding.embedding.length} dims): '
'${embedding.embedding.take(5)}...');
}
Texts are embedded as search queries. EmbeddingGemma prefixes queries and documents differently, so embed what you index as documents:
final documentVectors = await ai.embed(
embedder: flutterEdgeAi.embedder('embedding-gemma-300m'),
documents: chunks,
options: FlutterEdgeAiEmbedConfig(taskType: 'retrievalDocument'),
);
The embedder takes text only: a document with no text part is an
INVALID_ARGUMENT. It resolves the active embedder again for each request,
so it follows a model that core replaced or closed in between.
Routing with a Decider (experimental)
DeciderRouteClassifier.load lets a genkit_hybrid router
decide on the device, with an EmbeddingGemma 2 decider (or InferenceModel.createDecider), which
of its targets a request goes to. The decider's options are the labels; the policy maps them to
targets:
// ignore_for_file: experimental_member_use
final intents = DeciderRouteClassifier.load(() async {
final embedder = await FlutterEdgeAi.getActiveEmbedder();
return embedder.createDecider(options: [
DeciderOption('smalltalk', examples: [
EmbeddingInput.text('how are you today?'),
EmbeddingInput.text('tell me something nice'),
]),
DeciderOption('code', examples: [
EmbeddingInput.text('write a parser in Rust'),
EmbeddingInput.text('why does this SQL query time out?'),
]),
]);
}); // loading starts now
final router = defineHybridRouter(
ai, // a Genkit with GenkitFlutterEdgeAiPlugin and no default model
targets: {
kOnDevice: HybridTarget.onDevice(
flutterEdgeAi.model('gemma-4-e2b'),
warmUp: flutterEdgeAi.warmUp,
),
kCloud: HybridTarget.cloud(googleAI.gemini('gemini-2.5-flash')),
},
cloudPlugins: [googleAI(apiKey: apiKey)],
policies: {
'smart': RoutePolicy(
rules: [RouteRule.inputSize(maxChars: 3000, small: kOnDevice, large: kCloud)],
classifier: intents,
labels: {
'smalltalk': RouteDecision.prefer(kOnDevice),
'code': RouteDecision.prefer(kCloud),
},
),
},
defaultPolicy: 'smart',
);
await router.warmUp(); // the decider and the on-device model, before the first request
- The last user message's text is classified (or what
text:picks); none is an abstain (noText), which goes to the policy'sonAbstain(defaultRouteDecision.only(kOnDevice)). DeciderRouteClassifier.loadowns its decider: loading starts when it is created, and a load that fails fails the requests waiting for it (never routed around) and runs again at the next request.reload()refits: the old decider answers until the new one is ready, then is closed; a failed reload keeps it.close()closes the decider.- To keep the decider yourself,
deciderRouteClassifier(() => decider)asks your provider per request: return the pending Future while it loads (the request waits, with no limit unless the call has a Genkit cancel token), null when there is none (the classifier is unavailable — the call fails unless the policy setsonUnavailable). A Future that fails fails the call; drop it from your cache so the next request retries. This classifier never closes your decider. - A decider closed by a refit is asked for once more; one whose embedder was replaced under it
chooses once more. Still closed, the classifier is unavailable, and
route.reasonsays why. flutterEdgeAi.warmUp(ref)loads the active model with the ref's config, the arguments a request with that config uses, so the first request reuses it; an invalid config isINVALID_ARGUMENT.- Every other decider error fails the call, never an abstain — an input over the embedder's
limit included, so keep
RouteRule.inputSizebefore the classifier. - The route info carries the label, the decider's abstain signal as
score, the margin, and its method, calibration, space id and timings underdetails.
Built something with flutter_edge_ai?
See what others ship on the showcase, and add your app: it takes a couple of minutes, and the listing links to your app's Google Play or App Store page.
Known Limitations
- Abandoned streams: Genkit 1.0 does not tell a model action when a stream consumer goes away. Cancel through a
CancellationToken, or cap the reply withmaxOutputTokens; otherwise the generation finishes on its own and then closes its chat. - Engine registration: In the current
flutter_edge_aiarchitecture, inference engines and embedding backends are opt-in. The host app must add the relevant packages (flutter_edge_ai_litertlmfor.litertlm,flutter_edge_ai_mediapipefor.task/.bin,flutter_edge_ai_onnxfor ONNX,flutter_edge_ai_llamafor.gguf,flutter_edge_ai_builtin_aifor the OS's own model,flutter_edge_ai_embeddingsplus a backend such asflutter_edge_ai_litertlm'sLiteRtEmbeddingBackendfor embeddings) and register their providers inawait FlutterEdgeAi.initialize()before using the plugin. This split first shipped asflutter_gemma1.0.0. - Model installation: The plugin does NOT manage model installation. The host app must install models via
FlutterEdgeAi.installModel()and embedders viaFlutterEdgeAi.installEmbedder()before using the plugin. Without one, a request fails withUNAVAILABLE. - Model names are labels: every configured
FlutterEdgeAiModelConfigruns the model that is active in flutter_edge_ai, the one installed or selected last;modelTypetells the plugin how to talk to it. The names share one queue, so generations run one at a time in the order they arrive. Each request asks flutter_edge_ai for the active model, which reuses the loaded one while nothing changed; a newly installed model, or other runtime arguments, give the next request a new one. SetsupportsTools: falseorsupportsMedia: falseon a config whose model takes no tools or media: itssupportsmetadata says so, and a request that needs them fails withUNIMPLEMENTEDinstead of reaching a model that would ignore them. - System role: System messages are passed natively via
openChat(systemInstruction:). Only text content is supported in system messages. The capability first shipped under the old package name influtter_gemma0.13.0. - Web: media from
file://URIs and paths isUNIMPLEMENTED, ablob:URL is refused, andhttp(s)media needs CORS headers on its server. The web engines report no token counts, soresponse.usagecarries the media counts only. - Thinking mode: DeepSeek
.taskexposes thinking on Android/iOS; Gemma 4 exposes it through.litertlmengines, native and Web. Qwen3's emitted<think>tags are parsed by core platform-independently, including Web. MediaPipe Web has no thinking API, ONNX Web ignoresenableThinking, and the catalog has no DeepSeek Web entry.
Libraries
- genkit_flutter_edge_ai
- Genkit Dart plugin for flutter_edge_ai — local on-device AI inference.