llamadart 0.19.5

Cross-platform, on-device LLM inference for Dart and Flutter. Run GGUF (llama.cpp) and LiteRT-LM models locally on Android, iOS, macOS, Windows, Linux and web.

example/README.md

llamadart Examples

This directory contains example applications demonstrating how to use the llamadart package.

LFM2 automatic Python tool calls require native llama.cpp with lazy grammar and prompt penalties under grammar. Published Web runtimes reject that path before inference; see the tool-calling guide for typed legacy ToolParam declarations and the separate required/none paths.

Available Examples

1. Basic App (basic_app/)

A simple console application showing:

• Model loading
• Context creation
• Tokenization
• Native LiteRT-LM JSON Schema with Qwen3 or Gemma 4 (bin/llamadart_litert_json_example.dart)
• Text generation
• Embedding generation
• Decision-model answers with DecisionEngine
• Image generation with ImageGenerationEngine (bin/llamadart_image_example.dart)
• Resource cleanup

Best for: Understanding the core API

Run:

cd basic_app
dart pub get
dart run

# Embedding demo
dart run bin/llamadart_embedding_example.dart -i "hello world"

# SQLite vector retrieval demo
dart run bin/llamadart_sqlite_vector_example.dart \
  -q "How do I improve embedding throughput?" \
  -d "Increase maxParallelSequences for wider embedding batches." \
  -d "Tune batchSize and ubatchSize together."

# Decision model demo (Laya ticket triage)
dart run bin/llamadart_decision_example.dart --json

2. Chat App (chat_app/)

This app and its private validation harness run from a repository checkout, not the core pub archive. Clone the repository before following these commands.

A Flutter UI application showing:

• Real-time chat interface
• Model configuration
• Settings persistence
• Streaming text generation
• GGUF plus LiteRT-LM .litertlm model routing
• On-device image generation screen (SDXS, SD-Turbo)
• Material Design UI

Best for: Real-world Flutter integration

Run:

cd chat_app
flutter pub get
flutter run

For LiteRT-LM Web text chat, choose a browser-compatible bundle and check the loaded runtime before relying on history or system instructions:

final engine = await LlamaEngine.load(
  LlamaModel(ModelSource.parse('https://your-host/model-web.litertlm')),
  backend: LlamaBackend.withTextHistory(),
);
final capabilities = await engine.capabilities;
if (!capabilities.supportsMultiTurnChat) {
  throw LlamaUnsupportedException('This Web runtime cannot preserve history.');
}
final session = ChatSession(engine, systemPrompt: 'Answer briefly.');
await session.send('Remember PINE.');
final reply = await session.send('Which word did I ask you to remember?');
session.reset(keepSystemPrompt: false);

For a direct Web backend, opt in with LiteRtLmBackend.withTextHistory(). The portable LlamaBackend.withTextHistory() factory selects this path on Web and preserves normal native routing. Existing default/direct constructors keep single-user-text routing for LiteRT Web models, reporting multi-turn support false; custom/GGUF delegate capabilities remain unchanged.

This Web path forwards system/user/assistant text, and tools when the loaded runtime reads preface.tools (supportsToolCalling). Keep media, custom template/thinking controls, tool-result content, ToolChoice.required and parallelToolCalls out of the request; they raise LlamaUnsupportedException (#1152). Automatic tool loops run only when the load-time token-count probes pass; see Tool calling.

3. API Server (llamadart_server/)

A Relic-based HTTP server example showing:

• OpenAI-compatible endpoint surface (/v1/models, /v1/chat/completions)
• OpenAPI spec + Swagger UI (/openapi.json, /docs)
• SSE streaming responses for chat completions
• Optional Bearer auth and CORS middleware
• Model loading via local path or URL download

Best for: Local API integration with OpenAI-style clients

Run:

cd llamadart_server
dart pub get
dart run llamadart_server --model /path/to/model.gguf

4. llama.cpp-style CLI (llamadart_cli/)

A compatibility-focused CLI clone showing:

• llama.cpp-like options (--model, -hf, --ctx-size, --fit, --jinja)
• Interactive terminal chat with streaming output
• Hugging Face shorthand resolution for GGUF files
• GLM-oriented sampling controls for Unsloth command parity

Best for: Running llama.cpp-like local chat flows in pure Dart

Run:

cd llamadart_cli
dart pub get
dart run bin/llamadart_cli.dart --help

5. TUI coding agent (tui_coding_agent/)

A small, Pi-style terminal coding agent built with nocterm and llamadart showing:

• One sequential JSON tool loop with exactly read, write, edit, and bash
• A single-screen, single-session chat UI
• A compact TurboVision-inspired frame with streamed reasoning and dense Markdown answers, including syntax-highlighted fenced code blocks
• Workspace-confined file tools plus bounded, cancellable source downloads, generation, and active shell work; a model load cancelled during native allocation unloads after that allocation call returns
• An Unsloth Qwen3.6 35B-A3B UD-Q4_K_M default with publisher-aligned non-thinking sampling and shared llamadart model caching
• Optional --thinking mode with a larger context and coding-focused sampler
• Startup-only workspace and model selection
• Optional --read-only mode that exposes only the read tool

The bash tool uses Bash on Unix or cmd.exe on Windows and runs with the current user's permissions. It is not sandboxed, and file-tool workspace confinement does not apply to shell commands.

Best for: Building local terminal coding-agent workflows in Dart

Run:

cd tui_coding_agent
dart pub get
dart run bin/tui_coding_agent.dart

6. Laya Tetris (laya_tetris/)

A Flutter app (macOS, iOS, Android, Web) in which a Laya decision model plays real-time Tetris through DecisionEngine:

• One backbone GGUF in a LlamaEngine shared by two heads: the base head and a Tetris-tuned head
• Yes/no, choice, and knockout players, each piece's questions sent in one systemOneBatch call while gravity keeps running
• First-launch downloads of the pinned fr0stbit3/laya-gguf and leehack/laya-tetris-head files with progress, cached across launches
• bin/bench.dart for headless games with local model files, and bin/make_dataset.dart for the tuned head's training data

Best for: Decision models in a real-time Flutter app

Run:

cd laya_tetris
flutter pub get
flutter run -d macos

Web needs the WebGPU bridge assets and cross-origin isolation headers; see laya_tetris/README.md.

7. Laya Command Bar (laya_command_bar/)

A Flutter app (macOS, iOS, Android, Web) with one text field that reshapes as you type, read by one of four switchable readers:

• A Laya choice question through DecisionEngine, EmbeddingGemma nearest labelled examples, or two small LLMs' next-token scores through LlamaEngine.scoreNextToken: Qwen2.5 with examples, and the decision model decider-2b
• At most one read running, with the latest text read next
• A confidence gate that changes the bar only when the reader is sure, and keeps it steady between keystrokes
• Reminder, event, message, calculation, search, task, question and settings controls, filled by rule-based parsers
• bin/bench.dart to score a reader and its gate with local model files

Best for: Interactive UI driven by a decision model or a small LLM

Run:

cd laya_command_bar
flutter pub get
flutter run -d macos

Web needs the WebGPU bridge assets and cross-origin isolation headers; see laya_command_bar/README.md.

For LFM tool replies cut off at a reported length limit, inspect completion.partialToolCalls for raw unfinished-envelope diagnostics. Execute only completion.toolCalls; diagnostic arguments can be unfinished Python or JSON, and completion.message deliberately excludes them. See the tool-calling guide.

Testing

• basic_app (Dart console):

cd basic_app
dart test

• chat_app (Flutter UI):

cd chat_app
flutter test test test_vm

• llamadart_server (Relic HTTP API):

cd llamadart_server
dart test

• llamadart_cli (llama.cpp-style Dart CLI):

cd llamadart_cli
dart test

• tui_coding_agent (nocterm-based coding agent TUI):

cd tui_coding_agent
dart test

• laya_tetris (Flutter decision-model game):

cd laya_tetris
flutter test

• laya_command_bar (Flutter command bar):

cd laya_command_bar
flutter test

Note: chat_app, laya_tetris and laya_command_bar use Flutter libraries (dart:ui), so dart test is not the correct runner for those examples.

Quick Start

1. Choose an example: Basic (console), Chat (Flutter), API Server (Relic), llama.cpp-style CLI clone (Dart), TUI coding agent (Dart + nocterm), Laya Tetris (Flutter decision-model game), or Laya Command Bar (Flutter command bar)
2. Download a model (see each example's README)
3. Run the example: Follow instructions in each subdirectory

Testing pub.dev Package

These examples simulate how users will use llamadart when published to pub.dev:

• They add llamadart as a dependency
• They rely on automatic library download
• They don't need to run build scripts

Common Models for Testing

• TinyLlama (1.1B, ~638MB) - Fast, good for testing
• Llama 2 (7B, ~4GB) - More powerful, slower
• Mistral (7B, ~4GB) - Great performance

See HuggingFace for more: https://huggingface.co/models?search=gguf

Model Formats

Most examples use GGUF models through the llama.cpp runtime. The Flutter chat_app also demonstrates LiteRT-LM .litertlm models and explicitly opts into the litert_lm native runtime family for supported native targets.

Architecture

example/
├── basic_app/          # Console application
│   ├── lib/            # Dart code
│   ├── pubspec.yaml    # Dependencies
│   └── README.md       # Instructions
├── llamadart_server/   # OpenAI-compatible API server
│   ├── bin/            # Server entrypoint
│   ├── lib/            # Request/response mapping + middleware
│   ├── pubspec.yaml    # Dependencies
│   └── README.md       # Instructions
├── llamadart_cli/      # llama.cpp-style Dart CLI clone
│   ├── bin/            # CLI entrypoint
│   ├── lib/            # Parser + model resolver + chat runner
│   ├── pubspec.yaml    # Dependencies
│   └── README.md       # Instructions
├── tui_coding_agent/   # Nocterm-based coding agent TUI
│   ├── bin/            # TUI entrypoint
│   ├── lib/            # Session, tools, and TUI components
│   ├── pubspec.yaml    # Dependencies
│   └── README.md       # Instructions
├── laya_tetris/        # Flutter decision-model game
│   ├── bin/            # Headless bench + dataset tool
│   ├── lib/            # Game, players, and Laya loading
│   ├── pubspec.yaml    # Dependencies
│   └── README.md       # Instructions
├── laya_command_bar/   # Flutter command bar
│   ├── bin/            # Intent and gate benchmark
│   ├── lib/            # Bar, gate, runner, readers, and parsers
│   ├── pubspec.yaml    # Dependencies
│   └── README.md       # Instructions
└── chat_app/           # Flutter application
    ├── lib/            # Flutter code
    ├── android/        # Android config
    ├── ios/            # iOS config
    ├── pubspec.yaml    # Dependencies
    └── README.md       # Instructions

Need Help?

• Check individual example README files
• Report issues: https://github.com/leehack/llamadart/issues
• Docs: https://github.com/leehack/llamadart

Requirements

• Dart SDK 3.10.7 or higher
• For chat_app, laya_tetris and laya_command_bar: Flutter 3.38.6 or higher
• Flutter Apple builds require deployment targets of iOS 16.4 or newer and macOS 14.0 or newer
• Internet connection (for first run - downloads selected native runtime libraries)
• At least 2GB RAM minimum, 4GB+ recommended

Platform Compatibility

Platform	Architecture(s)	GPU Backend	Status
macOS	arm64, x86_64	Metal	✅ Tested
iOS	arm64 (Device), arm64/x86_64 (Sim)	Metal (Device), CPU (Sim)	✅ Tested
Android	arm64-v8a, x86_64	CPU; Vulkan (experimental, device-dependent)	✅ Tested
Linux	arm64, x86_64	Vulkan	🟡 Expected (Vulkan Untested)
Windows	x64	Vulkan	✅ Tested
Web	WASM / WebGPU Bridge	CPU / Experimental WebGPU	✅ Tested

Web Notes

• Web examples run on the llama.cpp bridge backend (WebGPU or CPU mode).
• chat_app loader prefers local bridge assets on localhost, and otherwise prefers CDN with local fallback.
• You can prefetch a pinned bridge version into web/webgpu_bridge/ with: WEBGPU_BRIDGE_ASSETS_TAG=<tag> ./scripts/fetch_webgpu_bridge_assets.sh.
• Web model loading uses browser Cache Storage by default, so repeated loads of the same model URL can skip full re-download.
• Safari WebGPU uses a compatibility gate in llamadart: legacy bridge assets default to CPU fallback, while adaptive bridge assets can probe/cap GPU layers and auto-fallback to CPU when unstable.
• You can still bypass the legacy safeguard with window.__llamadartAllowSafariWebGpu = true before model load.
• Multimodal projector loading works on web via URL-based model/mmproj pairs.
• In chat_app, image/audio attachments on web are sent as browser file bytes; local file paths are native-only.
• Native LoRA runtime adapter flows are not available on web.
• chat_app on web uses model URLs rather than native file download storage.

Troubleshooting

Common Issues:

1. Failed to load library:

   • Check console for download messages
   • Ensure internet connection for first run
   • Verify GitHub releases are accessible

2. Model file not found:

   • Download a model to the default location
   • Or set LLAMA_MODEL_PATH environment variable
   • Or configure in app settings (chat_app)

3. Slow performance:

   • Use smaller quantization (Q4_K_M recommended)
   • Reduce context size (nCtx parameter)
   • Enable GPU layers if available

4. Flutter build errors:

   • Ensure Flutter SDK is properly installed
   • Run flutter doctor to check setup
   • Reinstall dependencies with flutter clean && flutter pub get

Security Notes

• Models downloaded from the internet should be from trusted sources
• Never share private/sensitive data with open-source models
• The app runs locally - no data is sent to external servers (except library download on first run)

Contributing

To contribute new examples:

1. Create a new subdirectory in example/
2. Add a pubspec.yaml with llamadart as dependency
3. Include a README.md with setup instructions
4. Test on multiple platforms if possible
5. Add durable coverage to test/, tool/testing/run_local_e2e.dart, or tool/testing/test_matrix.dart when the example needs repeatable validation

License

These examples are part of the llamadart project and follow the same license.

For a renamed legacy Qwen3 text bundle on native LiteRT-LM, select the current canonical template through the public API:

await engine.loadModel(
  'models/download',
  modelParams: ModelParams(chatTemplate: liteRtLmQwen3ChatTemplate),
  format: ModelFormat.liteRtLm,
);
final reply = await engine.complete(
  [LlamaChatMessage.fromText(role: LlamaChatRole.user, text: 'Hello')],
  enableThinking: false,
);

Obtain the template from the accessor after every package upgrade. Web @litert-lm/core@0.19.0 cannot apply this override and throws LlamaUnsupportedException at load; this example requires native LiteRT-LM with the conversation-template setter.