genkit_llamadart 3.1.1

Genkit Dart plugin for local, on-device LLMs through llamadart: llama.cpp GGUF and LiteRT-LM chat, streaming, tool calling, JSON output, and embeddings.

genkit_llamadart

Run local LLMs in Genkit Dart applications. Use llama.cpp GGUF models or LiteRT-LM bundles through llamadart for offline, on-device AI in Dart and Flutter: chat, token streaming, tool calling, structured JSON output and text embeddings.

The model runs in your application's process. Download it once or supply a local file, then run inference offline. Features depend on the loaded model and backend; strict JSON requires grammar support and embeddings require an embedding model. See runtime support.

Install

Requires Dart 3.12 or newer, Genkit 1.x and llamadart 0.11.x.

dart pub add genkit genkit_llamadart

For Flutter apps, use flutter pub add instead. Follow the llamadart installation guide for runtime and platform setup. Apple apps using Swift Package Manager need the runtime companion matching their model family:

flutter pub add llamadart_llama_cpp_flutter
flutter pub add llamadart_litert_lm_flutter

Choose the GGUF companion, the LiteRT-LM companion, or both if your app uses both runtimes. Apple companion builds require iOS 16.4+ or macOS 14.0+; an iOS Podfile must also target 16.4+ when using CocoaPods.

llamadart supports Android, iOS, macOS, Linux, Windows and Web. Backend features and setup differ by platform. This adapter's browser guard tests do not qualify a complete Genkit Web deployment; validate your chosen browser and runtime.

Quickstart

Save this as bin/main.dart in your Dart app, then run dart run bin/main.dart. This native Dart example downloads Qwen3.5-0.8B Q4_K_M (about 533 MB) from Hugging Face; later runs reuse the cached model.

import 'package:genkit/genkit.dart';
import 'package:genkit_llamadart/genkit_llamadart.dart';

Future<void> main() async {
  final prepared = await llamaDart.prepareModel(
    name: 'local-chat',
    source: ModelSource.parse(
      'hf://unsloth/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q4_K_M.gguf',
    ),
    modelParams: const ModelParams(
      contextSize: 4096,
      device: ComputeDevice.cpu,
    ),
  );
  final ai = prepared.createGenkit();
  const config = LlamaDartGenerationConfig(
    temperature: 1.0,
    topP: 1.0,
    topK: 20,
    penalty: 1.0,
    presencePenalty: 2.0,
    maxTokens: 128,
  );

  try {
    final response = await ai.generate(
      prompt: 'Say hello in one sentence.',
      config: config,
    );
    if (response.finishReason == FinishReason.failed ||
        response.finishReason == FinishReason.aborted) {
      throw response.cause ?? StateError('Generation did not complete.');
    }
    print(response.text);
  } finally {
    await prepared.dispose();
    await ai.shutdown();
  }
}

The 4,096-token context and 128-token output limit bound this example. CPU execution needs no GPU setup on supported native platforms. Sampling follows Qwen's nonthinking text recommendations. This example runs text-only on native llama.cpp; image input needs a matching projector. Omitted minP and enableThinking default to 0 and false. Use ComputeDevice.auto to let llamadart select a backend for your platform.

Weights download during preparation and load into the runtime on first use. Requests to the same model run one at a time; different model names use separate runtimes. Dispose the prepared handle when your app is done, then shut down Genkit. Disposal is terminal.

For other models, prepare a source from a local path, HTTPS URL or Hugging Face repository. The preparation guide covers revision pinning, checksum verification and download progress. Use an instruct/chat GGUF for chat, an embedding GGUF for vectors, or a .litertlm bundle on a supported LiteRT-LM target. Check the model card for its license and memory requirements.

Stream a response

Replace the contents of the quickstart's try block with this streaming code. It reuses the prepared model and sampling configuration and awaits onResult for the final outcome:

final stream = ai.generateStream(
  prompt: 'List three uses for offline AI, one short phrase per line.',
  config: config,
);
await for (final chunk in stream) {
  print(chunk.text);
}
final response = await stream.onResult;
if (response.finishReason == FinishReason.failed ||
    response.finishReason == FinishReason.aborted) {
  throw response.cause ?? StateError('Generation did not complete.');
}

For a stop button, call prepared.cancelActiveGeneration(). For request cancellation, pass a Genkit CancellationController token. See lifecycle and cancellation.

What you can build

Feature	Requirements / guide
Offline chat and streaming	Instruct/chat model; streaming example
Tool calling and agent loops	Tool-capable model/runtime, object input schemas; tool calling
Structured JSON output	Grammar-capable backend; JSON guide
Text embeddings for search / RAG	Embedding model; embedding guide
Image and audio input	Compatible multimodal model and runtime, matching projector when required; media guide
Downloads, cache and progress UI	ModelSource and preparation tasks; model preparation
OpenTelemetry traces and metrics	Application-configured telemetry provider; observability

The plugin checks loaded runtime capabilities before each request. Definition flags permit features; they do not guarantee model support. Discovery stays lazy, so support booleans start at false until verified. Unsupported requests fail explicitly. See feature permissions and discovery.

Strict JSON with active tools is currently unsupported. LiteRT-LM does not provide this adapter's grammar-constrained JSON or embeddings. Tool-call quality varies by model; runtime support alone does not prove reliability. This plugin embeds text for RAG; retrieval and Genkit document augmentation remain application responsibilities.

Runnable examples

Run from a checkout of this repository:

LLAMADART_MODEL_PATH=/models/chat.gguf \
dart run example/genkit_llamadart_example.dart

Example	Shows
Streaming chat	Incremental text and the final response
Download and cache	ModelSource preparation; defaults to a tiny SmolLM2 GGUF
Preparation UI	Progress snapshots, cancellation and warm-up
Agent loop	Genkit tools; interactive when LLAMADART_PROMPT is unset
Structured JSON	Typed output and grammar-constrained streaming
Embeddings	Vector dimensions and sample values; use an embedding model
Observability	Correlated spans and token metrics; needs an OTLP collector

The chat and agent examples accept LLAMADART_MMPROJ_PATH for a matching projector. Preparation examples accept a source instead of a local path:

dart run -DLLAMADART_MODEL_SOURCE=hf://owner/repo/model.gguf \
  example/genkit_llamadart_source_prepare_example.dart

Guides and API reference

• Generation settings (including thinking, presence penalties and speculative decoding), tools, JSON, embeddings and cancellation
• Model preparation, cache policy and loading UI
• Runtime support, discovery and custom runtimes
• Telemetry and engine observers
• Migration to 3.0 / Genkit 1.0
• Public API reference
• Changelog

Coding agent setup

The package includes Dart package skills for generation, model preparation, tool calling and structured output. Install them into your coding agent's skills directory from your app's root:

dart run skills@ get

Help and contributing

Report bugs or request features in the issue tracker. CONTRIBUTING.md covers local checks, meaningful regression tests and releases; ARCHITECTURE.md describes package layering.