llamadart

llamadart is a Dart package for on-device inference. It runs GGUF models through llama.cpp and .litertlm bundles through LiteRT-LM, on Android, iOS, macOS, Linux, Windows and the web, behind one Dart API. Web support is experimental. Native targets need no network once the model is on the device.

Streaming chat and text generation, with structured JSON output.
Tool calling driven by the model's chat template.
Embeddings, reranking, and image and audio input with multimodal models.
Speech to text, text to speech, and decision models.
Image generation from text prompts on native platforms (Preview).
Runtime LoRA adapters, observability hooks and typed capability checks.

Choose direct llamadart, Genkit, or Flutter Edge AI for your application. Genkit is a separate package; the Flutter Edge AI connector is owned upstream and pins its own llamadart version.

Install

dependencies:
  llamadart: ^0.17.3

dart pub get
# or
flutter pub get

Requirements: Dart SDK >=3.10.7; for Flutter apps, Flutter SDK >=3.38.0, iOS deployment target 16.4 and macOS deployment target 14.0 or newer. Consumers need no local C++ toolchain: the package build hook resolves native runtime archives on first build or run. Windows and other per-platform notes: Installation.

Flutter iOS and macOS apps that link the Apple XCFrameworks through Swift Package Manager add the companion packages for the runtimes they use. A companion shares its core's MAJOR.MINOR and constrains llamadart to it.

dependencies:
  llamadart: ^0.17.3
  llamadart_llama_cpp_flutter: ^0.17.1 # GGUF / llama.cpp
  llamadart_litert_lm_flutter: ^0.17.1 # Apple .litertlm / LiteRT-LM targets
  llamadart_stable_diffusion_flutter: ^0.17.1 # Apple image generation, opt-in

Pairing rules, the SwiftPM bootstrap and the privacy manifest are in Installation and Apple setup. Web apps run dart run llamadart:web_setup once; see the WebGPU bridge.

llamadart also ships agent skills that teach coding agents its APIs. Install them from your app's root:

dart run skills@ get

To check a desktop setup without a model, run dart run llamadart:doctor; see setup report scope.

First generation

import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final engine = await LlamaEngine.load(
    LlamaModel(
      ModelSource.parse(
        'hf://unsloth/SmolLM2-135M-Instruct-GGUF/'
        'SmolLM2-135M-Instruct-Q2_K.gguf',
      ),
    ),
  );

  try {
    final reply = await engine.create(
      const [
        LlamaChatMessage.fromText(
          role: LlamaChatRole.user,
          text: 'Explain local inference in one sentence.',
        ),
      ],
      params: const GenerationParams(maxTokens: 48),
    ).text();
    print(reply);
  } finally {
    await engine.dispose();
  }
}

Dispose every engine before a native host calls C exit(); see Model lifecycle. For multi-turn chat, wrap the engine in ChatSession: First chat session.

Runtimes and platforms

LlamaBackend() routes by model file type. After a model loads, engine.runtime names the runtime and await engine.capabilities reports what it supports, so branch on those instead of the file extension. Unsupported runtime and option combinations are rejected with LlamaUnsupportedException instead of silently degrading.

Model format	Typical use	Runtime
GGUF	Broad llama.cpp compatibility, Metal/Vulkan/CUDA/CPU, WebGPU bridge	llama.cpp
`.litertlm`	LiteRT-LM deployments, Android GPU/NPU-oriented bundles, Gemma-family LiteRT packages	LiteRT-LM

llama.cpp uses Metal on Apple platforms, Vulkan on Linux and Windows, and WebGPU in the browser. Android Vulkan is experimental and opt-in. LiteRT-LM can also use GPU and, on compatible Android deployments, NPU. Image generation is not available on the web. Per-target support, device limitations and runtime options:

Current default runtime pins:

Runtime	Pin
Native llama.cpp / GGUF	`leehack/llamadart-native@v0.8.0-1`
Native LiteRT-LM / `.litertlm`	`leehack/litert-lm-native@v0.18.1-4`
Web llama.cpp / GGUF	`leehack/llama-web-bridge-assets@v0.1.79`
Web LiteRT-LM / `.litertlm`	`@litert-lm/core@0.18.0`

Native overrides accept stable vMAJOR.MINOR.PATCH releases and explicit historical or nightly bNNNN artifacts. New nightly wrapper rebuilds use bNNNN-N; existing bNNNN-llamadart.N artifacts stay valid as consumption-only overrides. Nightly cores use canonical decimal spelling, and rebuild counters start at 1 without leading zeros. Build-hook overrides must always name an explicit tag; latest is limited to maintainer synchronization and header/binding regeneration. On Windows, runtime environment directory overrides select backend modules; use llamadart_native_path to replace the complete runtime. The full override rules are in Native runtime configuration.

Guides

Task	Docs
Resolve local paths, URLs, and Hugging Face sources	Finding models
Stream text and collect output	Generation and streaming
Generate typed JSON	Structured output
Use tool calling	Tool calling
Use images, audio, or projectors	Multimodal
Transcribe speech on device	Speech to text
Synthesize speech on device	Text to speech
Generate images on device (Preview)	Image generation
Answer typed questions with a decision model	Decision models
Generate embeddings and rerank	Embeddings
Load LoRA adapters	LoRA adapters
Save and restore KV state, write a custom backend	API levels
Trace and measure requests	Observability
Run Flutter Web / WebGPU	WebGPU bridge
Tune performance	Performance tuning
Upgrade from 0.16.x	0.17 migration guide

Examples

Run the Flutter chat app from a repository checkout, or try the hosted chat app.

Contributing

Repository checks, validation commands, and PR expectations are in AGENTS.md and the maintainer docs.

License

MIT. See LICENSE.