genkit_llamadart 3.1.0

Genkit Dart plugin for local, on-device LLMs through llamadart: llama.cpp GGUF and LiteRT-LM chat, streaming, tool calling, JSON output, and embeddings.

genkit_llamadart

Run local LLMs in Genkit Dart applications. Use llama.cpp GGUF models or LiteRT-LM bundles through llamadart for offline, on-device AI in Dart and Flutter: chat, token streaming, tool calling, structured JSON output and text embeddings.

The model runs in your application's process. Download it once or supply a local file, then run inference offline. Features depend on the loaded model and backend; strict JSON requires grammar support and embeddings require an embedding model. See runtime support.

Install

Requires Dart 3.12 or newer, Genkit 1.x and llamadart 0.11.x.

dart pub add genkit genkit_llamadart

For Flutter apps, use flutter pub add instead. Follow the llamadart installation guide for runtime and platform setup. Apple apps using Swift Package Manager need the runtime companion matching their model family:

flutter pub add llamadart_llama_cpp_flutter
flutter pub add llamadart_litert_lm_flutter

Choose the GGUF companion, the LiteRT-LM companion, or both if your app uses both runtimes. Apple companion builds require iOS 16.4+ or macOS 14.0+; an iOS Podfile must also target 16.4+ when using CocoaPods.

llamadart supports Android, iOS, macOS, Linux, Windows and Web. Backend features and setup differ by platform. This adapter's browser guard tests do not qualify a complete Genkit Web deployment; validate your chosen browser and runtime.

Quickstart

Replace /models/chat.gguf with your local instruct/chat model path.

import 'package:genkit/genkit.dart';

import 'package:genkit_llamadart/genkit_llamadart.dart';

Future<void> main() async {
  final plugin = llamaDart(
    models: const <LlamaModelDefinition>[
      LlamaModelDefinition(
        name: 'local-chat',
        modelPath: '/models/chat.gguf',
        supportsEmbeddings: false,
      ),
    ],
  );
  final ai = Genkit(plugins: <LlamaDartPlugin>[plugin]);

  try {
    final response = await ai.generate(
      model: llamaDart.model('local-chat'),
      prompt: 'Say hello in one sentence.',
      config: const LlamaDartGenerationConfig(
        temperature: 0.2,
        maxTokens: 96,
      ),
    );
    if (response.finishReason == FinishReason.failed ||
        response.finishReason == FinishReason.aborted) {
      throw response.cause ?? StateError('Generation did not complete.');
    }
    print(response.text);
  } finally {
    await plugin.dispose();
    await ai.shutdown();
  }
}

Models load on first use. Requests to the same model run one at a time; different model names use separate runtimes. Dispose the plugin when your app is done, then shut down Genkit. Disposal is terminal.

Don't have a model file? Prepare a model source from a local path, HTTPS URL or Hugging Face repository with caching, checksum verification and download progress. Use an instruct/chat GGUF for chat, an embedding GGUF for vectors, or a .litertlm bundle on a supported LiteRT-LM target. Check the model card for its license and memory requirements.

Stream a response

With the ai instance from the quickstart, use generateStream for incremental text and await onResult for the final outcome:

final stream = ai.generateStream<Object?, LlamaDartGenerationConfig>(
  model: llamaDart.model('local-chat'),
  prompt: 'Name three uses for offline AI.',
  config: const LlamaDartGenerationConfig(maxTokens: 128),
);
await for (final chunk in stream) {
  print(chunk.text);
}
final response = await stream.onResult;
if (response.finishReason == FinishReason.failed ||
    response.finishReason == FinishReason.aborted) {
  throw response.cause ?? StateError('Generation did not complete.');
}

For a stop button, call plugin.cancelActiveGeneration('local-chat'). For request cancellation, pass a Genkit CancellationController token. See lifecycle and cancellation.

What you can build

Feature	Requirements / guide
Offline chat and streaming	Instruct/chat model; streaming example
Tool calling and agent loops	Tool-capable model/runtime, object input schemas; tool calling
Structured JSON output	Grammar-capable backend; JSON guide
Text embeddings for search / RAG	Embedding model; embedding guide
Image and audio input	Compatible multimodal model and runtime, matching projector when required; media guide
Downloads, cache and progress UI	ModelSource and preparation tasks; model preparation
OpenTelemetry traces and metrics	Application-configured telemetry provider; observability

The plugin checks loaded runtime capabilities before each request. Definition flags permit features; they do not guarantee model support. Discovery stays lazy, so support booleans start at false until verified. Unsupported requests fail explicitly. See feature permissions and discovery.

Strict JSON with active tools is currently unsupported. LiteRT-LM does not provide this adapter's grammar-constrained JSON or embeddings. Tool-call quality varies by model; runtime support alone does not prove reliability. This plugin embeds text for RAG; retrieval and Genkit document augmentation remain application responsibilities.

Runnable examples

Run from a checkout of this repository:

LLAMADART_MODEL_PATH=/models/chat.gguf \
dart run example/genkit_llamadart_example.dart

Example	Shows
Streaming chat	Incremental text and the final response
Download and cache	ModelSource preparation; defaults to a tiny SmolLM2 GGUF
Preparation UI	Progress snapshots, cancellation and warm-up
Agent loop	Genkit tools; interactive when LLAMADART_PROMPT is unset
Structured JSON	Typed output and grammar-constrained streaming
Embeddings	Vector dimensions and sample values; use an embedding model
Observability	Correlated spans and token metrics; needs an OTLP collector

The chat and agent examples accept LLAMADART_MMPROJ_PATH for a matching projector. Preparation examples accept a source instead of a local path:

dart run -DLLAMADART_MODEL_SOURCE=hf://owner/repo/model.gguf \
  example/genkit_llamadart_source_prepare_example.dart

Guides and API reference

• Generation settings, tools, JSON, embeddings and cancellation
• Model preparation, cache policy and loading UI
• Runtime support, discovery and custom runtimes
• Telemetry and engine observers
• Migration to 3.0 / Genkit 1.0
• Public API reference
• Changelog

Coding agent setup

The package includes Dart package skills for generation, model preparation, tool calling and structured output. Install them into your coding agent's skills directory from your app's root:

dart run skills@ get

Help and contributing

Report bugs or request features in the issue tracker. CONTRIBUTING.md covers local checks, meaningful regression tests and releases; ARCHITECTURE.md describes package layering.