llama_cpp_flutter 0.5.0

On-device LLM inference for Flutter: llama.cpp with Metal on iOS/macOS, wllama on web. Streaming sessions, chat formats, model downloads, and optional agents integration.

Changelog

0.5.0

• New ArtifactStore, exported from package:llama_cpp_flutter: app-managed storage for model artifacts, with one API on every platform — a directory the app names on iOS/macOS, the origin-private file system on web. Build one with createArtifactStore, then fetch a URL or importFile / importStream a local file into it, and resolve the stored keys into the localPath / localMmprojPath / localDraftPath arguments loadModel takes. list, lookup, totalSizeBytes, and delete are what a model-library UI needs to show users what is on their device and take it back off.

  This is the alternative to an engine's opaque cache. Previously a host app could either hand loadModel a path it managed entirely itself (native only) or let the runtime download to somewhere it could not enumerate — on web, into wllama's internal URL cache.

  Downloads resume where they stopped and never report a partial file as complete: native transfers finalize through ModelDownloader's .part rename, and web transfers carry an in-flight marker alongside the data. importFile copies by default — a macOS document picker hands back the user's real path — and moves only when the caller opts in with moveSource, renaming instead of duplicating a multi-gigabyte GGUF when the source is on the same volume. Storage failures surface as ArtifactStorageException, with isQuotaExceeded set when the browser is out of room.

  On web, resolve returns opaque handles rather than blob URLs, and the runtime dereferences them straight to their stored files. A model past the ~2 GiB wasm32 per-file limit is split into stageable parts from disk, as the runtime's own oversized-model path already did — so managed storage handles large models transparently instead of reading gigabytes back through the network stack. Speculative-decoding draft models stay native-only.

• The web runtime's oversized-model cache now writes through the same OPFS directory and names as ArtifactStore, so a model it downloaded on its own is listable and deletable through managed storage. Artifacts cached by earlier versions are adopted in place, not re-downloaded.

• package:llama_cpp_flutter/gguf.dart adds importedArtifactFileName and artifactDisplayName alongside stableArtifactFileName.

• Example: the hidden model cache is now a user-managed model library built on ArtifactStore. Models are added from the catalog, from a download URL, or from GGUF files on the device — each with an optional vision projector and, on native, a draft model — and the library shows what is stored, what it costs, and which model is loaded. Entries are editable (save, or save and reload), deletion is confirmed and removes only files no other entry references, and the catalog is persisted and reconciled with storage at startup. Image attachments are enabled only while the loaded model has a projector.

0.4.0

• New TokenSmoother stream transformer and the Stream<String>.smoothed() extension, exported from package:llama_cpp_flutter/chat.dart. Token streams arrive in bursts; the smoother re-paces them into a steady, typewriter-style grapheme stream for display.

  The release rate tracks the measured arrival rate rather than draining the backlog on a fixed schedule — it estimates how fast text is arriving, releases at that rate while holding about window in reserve, and uses the backlog only as a correction, with the rate low-pass filtered over smoothing so speed changes ramp instead of jump. The rate is fractional and carried across frames, so it can emit slower than one grapheme per frame and match a slow model instead of outrunning it. Against a 5 tok/s source the worst gap between graphemes is 64 ms (median 48 ms), versus 152 ms (median 16 ms) for fixed-window draining, which empties its buffer in four frames and then stalls.

  Grapheme clusters (emoji ZWJ sequences) are never split, an atomic predicate keeps in-band markers from rendering half-formed, and the tail drains within window of the source ending. Opt-in by design — generation is not smoothed automatically, so headless and batch callers are unaffected. Apply it where the text is rendered: session.generate(prompt).smoothed().

• Adds a direct dependency on package:characters.

0.3.2

• Example: the web demo now deploys to GitHub Pages on every push to main, and vendors coi-serviceworker to inject COOP/COEP headers on hosts that can't set them — keeping wllama multi-threaded on Pages. No library changes.

0.3.1

• Upstream: llama.cpp b10069 → b10091 on both backends. The native xcframework re-pins to the b10091 release, and the vendored wasm is now built from wllama 3.5.1 sources with its llama.cpp submodule at that same tag (previously the stock npm build, whose embedded llama.cpp dated from June 2026). Keep loading @wllama/wllama 3.5.1 on the page — the JS pairing is unchanged. Verified in-browser: model download, load, and streamed generation.
• New maintenance pipeline: tool/update_deps.sh (native re-pin with ABI check), tool/update_wllama.sh (wasm refresh with recorded provenance), tool/check_llama_ext_abi.sh (staging-ABI tripwire), a CI workflow (analyze/test, ABI check, macOS link build), and a weekly workflow that re-pins both backends to the same llama.cpp release and opens a PR — including rebuilding the wllama wasm against the pinned tag.

0.3.0

• New: generateEvents() on LlamaSession — a typed Stream<LlamaGenerationEvent> (LlamaTextEvent / LlamaCompletedEvent / LlamaWarningEvent) over the same generation contract as generate(). A successful run always ends with a completed event carrying the engine's stats, so finish reason and token accounting cannot be missed. Non-breaking: implemented as an extension over the existing string stream.
• Fixed: the native session now reports capabilities.canSetImageTokenBudget as false for sessions loaded without a multimodal projector (previously always true, and calling setImageTokenBudget on a text-only session could only fail natively).

0.2.0

Pre-1.0 API cleanup release. Pinned upstream: llama.cpp b10069 (xcframework, sha256-verified); wllama wasm vendored as a Flutter asset.

Breaking changes

• The bridge session type is renamed LlamaSession → LlamaBridgeSession (bridge.dart); the neutral LlamaSession in the main entrypoint is unchanged.
• LlamaChatTurn now carries a typed role (LlamaChatRole, including tool) and ordered content parts (LlamaContentPart: LlamaTextPart / LlamaImagePart / LlamaAudioPart) instead of a role string with separate text/images/audio fields. The old fields remain as read-only getters derived from parts; construct turns with parts: or LlamaChatTurn.text(...). Interleaved text/media ordering is now preserved end to end (the web runtime sends parts in author order).
• The main entrypoint exports far less. Concrete chat formats, templates, and stream decoders moved to package:llama_cpp_flutter/chat.dart; GGUF metadata reading and artifact cache naming to package:llama_cpp_flutter/gguf.dart; the orchestration layer to package:llama_cpp_flutter/orchestration.dart. resolveChatFormat, detectChatFormatNameForGguf, createLlamaChatClient, ModelSpec, ModelDownloader, and the runtime API stay in the main entrypoint.
• Overriding the pinned xcframework tag (LLAMA_CPP_TAG_OVERRIDE) now requires the matching LLAMA_XCFRAMEWORK_ZIP_SHA256_OVERRIDE, or an explicit LLAMA_CPP_ALLOW_UNVERIFIED=1 opt-out (development only); previously verification was silently skipped.

Changed

• SDK floor lowered from Dart ^3.12.0 / Flutter >=3.44.0 to Dart ^3.11.5 / Flutter >=3.41.7 (the floor now comes from the agents dependency, not this package).
• createLlamaRuntime() gains artifactCacheDirectory for download-on-load, and throws a clear UnsupportedError on IO platforms without a native backend (Android, Windows, Linux) instead of failing later with a missing-plugin error.
• LlamaSession documents its concurrency and lifecycle contract (one generation at a time; overlapping generate supersedes; cancel and dispose-during-generation semantics).
• Core data classes (ModelSpec, SamplingDefaults, LlamaGenerationStats, LlamaSessionCapabilities, ImageTiling, AgentSnapshot) gain value equality and toString; LlamaSessionCapabilities and ModelSpec assert their invariants.

0.1.0

Initial release.

• Cross-platform LlamaRuntime / LlamaSession API over llama.cpp:
  • iOS / macOS via a vendored Metal-enabled llama.xcframework (downloaded from official llama.cpp releases during pod install) with a typed Pigeon bridge. All native calls run on a dedicated worker isolate so model loading and token streaming never block the UI.
  • Web via @wllama/wllama, with the matching wllama.wasm shipped as a Flutter asset.
• ChatClient implementations for the agents framework, backed by local GGUF models.
• Chat format layer with auto-detection from GGUF metadata: ChatML, Gemma, LFM2, Llama 3, Mistral, and Qwen templates, including streaming tool-call decoding (Hermes-style and model-specific markers).
• GGUF utilities: metadata reader, split-model (-00001-of-0000N) support, and a resumable Hugging Face model downloader.
• Memory-aware multi-agent orchestration over a single loaded model: memory estimation and budget planning, runtime memory monitoring, session snapshot store, and artifact staging.
• MTP speculative decoding on iOS/macOS via the LlamaExtShim staging ABI, with a CI ABI-drift check.
• Prompt inspector diagnostics for examining rendered prompts.