fonnx 1.1.0

ONNX model inference for Dart and Flutter on every platform: ONNX Runtime through FFI on native, Web Workers on the web. Embeddings, Whisper, Kokoro TTS, and more.

Changelog

2026 Sep 29

• Moved the Potion 32M tokenizer into an opt-in library so 8M-only apps no longer load the 32M vocabulary in debug builds. Replace MinishLab.potion32mTokenizer with potion32mTokenizer and import package:fonnx/tokenizers/potion_32m_tokenizer.dart. MinishLab.potion8mTokenizer is unchanged.

2026 Aug 24

• Pronunciation transcription can preserve exact decoder token IDs, identify their vocabulary, decode them without re-tokenization, and feed compatible token sequences directly back into keyword matching.

2026 Aug 9

• Added runtime-configurable English keyword spotting on Android, iOS, Linux, macOS, Windows, and Web.
• All native targets use the same Dart FFI worker through Native Assets; no Kotlin, Swift, CocoaPods, Maven, or model-specific native runner is needed.
• Added runtime phrase replacement, pronunciation aliases, teach-by-voice transcription, streaming fixtures, and Android/iOS integration tests.
• Added the ORT 1.27.0 Apple SME Zipformer workaround. It is scoped to the encoder session and can be removed after upgrading the Native Asset to fixed ORT 1.27.1 or newer.

2026 Jul 20

• Upgraded ONNX Runtime from 1.16.1 to 1.27.0 on every native platform.
• Replaced vendored binaries, CocoaPods, Maven runtime dependencies, and Kotlin/Swift platform channels with SHA-256-verified native code assets and one Dart FFI inference implementation.
• Added Linux aarch64 support for Raspberry Pi/flutter-pi.
• Removed ONNX Runtime Extensions. Every model runs on core ORT; tokenizers and token decoding are in Dart.
• Native support is Android armv7/arm64/x64, iOS arm64 device/simulator, Linux x64/aarch64, macOS arm64, and Windows x64/arm64. Intel macOS is dropped.

2026 Jul 19

• Upgraded voice activity detection to the official Silero VAD v6.2.1 model.
• Added the v6 streaming state/context contract on native, mobile, and web.

2024 Apr 22

• CI builds for all platforms are now running on Codemagic..

2024 Feb 26

• Google's Magika for file identification supported on all platforms.
• Example app including full voice assistant flow, with Whisper, Silero voice activity detection. Available at telosnex.github.io/fonnx/

2024 Feb 19

• Whisper supported on all platforms, including web.

2024 Feb 13

• Whisper now supported on all platforms besides web.
• Whisper models support timestamps. (not exposed via API, yet)
• Silero VAD added to all platforms besides web.
• Silero VAD enables detecting when the user is done speaking with a much higher success rate than relying on volume levels.
• Example contains SttService, an example of how to integrate the VAD and Whisper together with an easy to use interface. (Stream

FONNX

Any model on any edge

Run ML models natively on any platform. ONNX models can be run on iOS, Android, Web, Linux, Windows, and macOS.

What is FONNX?

FONNX is a Flutter library for running ONNX models. Flutter, and FONNX, run natively on iOS, Android, Web, Linux, Windows, and macOS. FONNX leverages ONNX to provide native acceleration capabilities, from CoreML on iOS, to Android Neural Networks API on Android, to WASM SIMD on Web. Most models can be easily converted to ONNX format, including models from Pytorch, Tensorflow, and more.

Getting ONNX Models

Hugging Face

🤗 Hugging Face has a large collection of models, including many that are ONNX format. 90% of the models are Pytorch, which can be converted to ONNX.

Here is a search for ONNX models.

Export ONNX from Pytorch, Tensorflow, & more

A command-line tool called optimum-cli from HuggingFace converts Pytorch and Tensorflow models. This covers the vast majority of models. optimum-cli can also quantize models, significantly reduce model size, usually with negligible impact on accuracy.

See official documentation or the quick start snippet on GitHub.
Another tool that automates conversion to ONNX is HFOnnx. It was used to export the text embeddings models in this repo. Its advantages included a significantly smaller model size, and incorporating post-processing (pooling) into the model itself.

• Brief intro to how ONNX model format & runtime work huggingface.com
• Netron allows you to view ONNX models, inspect their runtime graph, and export them to other formats

FUTO keyboard models

FutoSwipe provides offline English QWERTY swipe decoding, next-word prediction, and ContextLM word scoring from the FUTO Swipe models:

final swipe = await FutoSwipe.load(
  bundle: FutoSwipeBundle.fromDirectory(modelDirectory),
  lexicon: const [FutoSwipeWord('hello', frequency: 200)],
);
final candidates = await swipe.decode(points, previousWords: const ['say']);

lib/models/futo_swipe/README.md documents the API contract, pipeline, and validation. The Web requires futo_swipe_init.js and futo_swipe_worker.js.

The converted bundle is in example/assets/models/futoSwipe/. It has the swipe encoder, the English QWERTY decoder, ContextLM, and get_embeddings.onnx. All graphs use core ONNX opset 17 and need no custom operators. manifest.json records the tensor contracts, source revisions, and hashes. The vocabulary, scoring configuration, metadata, license, and NOTICE.txt are distributed with the models. The artifact gate verifies every file.

The bundle is derived from FUTO Swipe and is subject to the FUTO Model Weights License 1.0 in LICENSE-FUTO.txt. Products must display visible FUTO Swipe attribution. Neither the conversion nor the runtime uses the GPL swipe library.

To reproduce the bundle, use FlatBuffers flatc 25.2.10 and an empty output directory:

dart run tool/futo/convert.dart /tmp/futo-swipe-onnx /path/to/flatc

The Dart converter translates the pinned .pte graphs and copies their FP32 weights. It needs no Python, training checkpoint, or ExecuTorch runtime. Inference is not bit-exact with ExecuTorch, especially near sigmoid saturation. test/data/futo_swipe/validation.json records the one-off numerical comparison. The network-enabled replay in tool/futo/replay.dart sends 1,000 raw public swipes through FutoSwipe.decode. It checks the results against frozen ExecuTorch top-three candidates. All FUTO-specific tools are in tool/futo/.

Text Embeddings

These models generate embeddings for text. An embedding is a vector of floating point numbers that represents the meaning of the text.
Embeddings are the foundation of a vector database, as well as retrieval augmented generation - deciding which text snippets to provide in the limited context window of an LLM like GPT.

Running locally using FONNX provides significant privacy benefits, as well as latency benefits. For example, rather than having to store the embedding and text of each chunk of a document on a server, they can be stored on-device. Both MiniLM L6 V2 and MSMARCO MiniLM L6 V3 are both the product of the Sentence Transformers project. Their website has excellent documentation explaining, for instance, semantic search

MiniLM L6 V2

Trained on a billion sentence pairs from diverse sources, from Reddit to WikiAnswers to StackExchange. MiniLM L6 V2 is well-suited for numerous tasks, from text classification to semantic search. It is optimized for symmetric search, where text is roughly of the same length and meaning. Input text is divided into approximately 200 words, and an embedding is generated for each.
🤗 Hugging Face

MSMARCO MiniLM L6 V3

Trained on pairs of Bing search queries to web pages that contained answers for the query. It is optimized for asymmetric semantic search, matching a search query to an answer. Additionally, it has 2x the input size of MiniLM L6 V2: it can accept up to 400 words as input for one embedding.
🤗 Hugging Face

Benchmarks

iPhone 14: 67 ms
Pixel Fold: 33 ms
macOS: 13 ms
WASM SIMD: 41 ms

Avg. ms for 1 Mini LM L6 V2 embedding / 200 words.

• Run on Thurs Oct 12th 2023.
• macOS and WASM-SIMD runs on MacBook Pro M2 Max.
• Average of 100 embeddings, after a warmup of 10.
• Input is mix of lorem ipsum text from 8 languages.

Prompt injection detection

WolfDefender runs Wolf Defender Edge by Patronus (80M parameters, Apache-2.0). It gives the probability that a text, such as an email, a web page or a tool result, has a prompt injection or a jailbreak. It was trained mainly on English and German.

final defender = WolfDefender.load(
  WolfDefenderBundle.inDirectory(modelDirectory),
);
final result = await defender.classify(emailText);
if (result.isInjection) { /* ... */ }

The model file is Patronus's weight-only int8 export (58 MB), with the tokenizer wolf_defender_edge_tokenizer.bin (3 MB) next to it. Both are runtime downloads (wolfDefender/ in runtimeModelNamesByPath). On 691 texts, its decisions matched the fp32 model on 99% on ARM, x86 and WebAssembly. fonnx does not use Patronus's other 58 MB file, with dynamic int8 activations: it matched on 98.7% on Apple M4, but on 94% on x86 and WebAssembly, and it reversed a reference decision on the ARM CPUs of CI.

A text of any length is split into windows of 2046 tokens that overlap by 64. On an M4 Max, a 72-token email takes 9 ms, and a full window 370 ms. The model needs about 200 MB after loading, and a full window about 600 MB more for the attention activations.

Integrating FONNX

Open-vocabulary keyword spotting

The 5 MB GigaSpeech Zipformer bundle detects English phrases selected at runtime. It accepts streaming mono 16 kHz PCM, supports atomic phrase changes, and can learn pronunciation aliases from a recording. Native targets share one Dart FFI implementation; Web uses an ONNX Runtime worker with the same Dart frontend and decoder.

final spotter = await KeywordSpotter.load(
  bundle: KeywordSpotterBundle.gigaSpeech3m(modelDirectory),
  keywords: const [
    KeywordPhrase('telosnex', spokenForms: ['tell us next']),
  ],
);
spotter.detections.listen((event) => print(event.phrase));
await spotter.acceptPcm16(microphoneBytes);
await spotter.close();

See the keyword spotter documentation and the example's realtime phrase editor.

Native platforms via Dart FFI and code assets

Android, iOS, Linux, macOS, and Windows use the same Dart FFI implementation. hook/build.dart uses package:kiln and native_artifacts/prebuilt.json. If the package sources match the release, it downloads and checks two bundled code assets: ONNX Runtime and the session finalizer. No C compiler is needed for a released package. Microsoft supplies the pinned ORT archives except on iOS, where FONNX supplies its existing dynamic ORT base. FONNX release CI builds the finalizer through the same hook that runs local source builds.

The native_build user define selects auto (default), download, or source. In auto mode, a source change makes the hook compile the finalizer. ORT remains a pinned upstream input in both modes. Source builds need CMake and a target compiler. Apple targets build on macOS, Linux on Linux, Windows on Windows, and Android with the pinned NDK.

hooks:
  user_defines:
    fonnx:
      native_build: source

There are no FONNX CocoaPods/Gradle native dependencies and no platform channels.

Microsoft's published iOS artifact is static and cannot be loaded as a Dart code asset. FONNX release CI runs Microsoft's official Apple framework script in its supported dynamic mode and publishes arm64 device/simulator artifacts for the hook. iOS 15.1 or newer is required. macOS 14 or newer and Apple Silicon are required; Intel support was intentionally dropped. Linux releases build on Ubuntu 22.04 and run with glibc 2.35 and its libstdc++. Windows requires the Microsoft Visual C++ 2015–2022 runtime. The package profile records these requirements.

Flutter reports iOS 13 and macOS 13 to hooks regardless of the app's target. FONNX uses its declared iOS 15.1/macOS 14 floors for prebuilt selection. The manifest keeps the real minima. Compiling from source cannot lower the minimum of the pinned ORT input.

Required iOS project setup

Set both the Runner deployment target and platform :ios in the Podfile to 15.1 or newer. Flutter 3.44.2's iOS native-assets pipeline generates framework Info.plist files with Flutter's iOS 13 baseline. It does not currently derive that value from a consuming Runner project's higher deployment target. The related upstream history is flutter/flutter#148044, and Flutter's source TODO points to the still-open flutter/flutter#145104.

ORT 1.27's Apple Mach-O binaries require iOS 15.1. App Store Connect rejects a framework that advertises a lower minimum than its binary with ITMS-90208. Until Flutter propagates the app target into native-asset packaging, add a Run Script phase immediately after Flutter's Thin Binary phase. The phase must set MinimumOSVersion on onnxruntime.framework to IPHONEOS_DEPLOYMENT_TARGET, verify that this minimum is not lower than the embedded Mach-O minimum, and re-sign the modified framework. The complete fail-closed implementation is in example/ios/fix_fonnx_framework_minimum_os.sh, and the example Xcode project shows the required phase ordering.

Runtime model downloads

The model files are on Hugging Face in telosnex/fonnx, not in Git. The generated prebuiltRuntimeFiles catalog pins the 62 model files: each URL names a Hugging Face commit, and the catalog has each file's size and SHA-256. Apps download a model after install:

import 'package:fonnx/runtime_models.dart';

final model = await ensureFonnxRuntimeModel(
  'pyannote/pyannote_seg3.onnx',
  modelDirectory,
);

The helper checks an existing file before reuse. It checks each download before it puts the file at its final path. The FUTO model names are distinct on Hugging Face and map back to their source paths through runtimeModelNamesByPath. Tokenizers, FUTO metadata, and license notices are separate from the ONNX model files.

The tests and the example app read the models in example/assets/models. To download them there, run:

dart run tool/fetch_hf_models.dart            # all models, 1.8 GB
dart run tool/fetch_hf_models.dart whisper    # one directory

To add or change a model, put the file in example/assets/models, add its entry without a commit to the models section of native_artifacts/manifest.json and to runtimeModelNamesByPath, then run dart run tool/publish_models.dart. It uploads the new files in one Hugging Face commit with hf upload, records the commit, and writes the catalog. A published file never changes. doc/ADR_001_MODELS_ON_HUGGING_FACE.md has the design.

To publish the native libraries and the web files, push to kiln-release, then merge the kiln-manifest/<tag> branch that .github/workflows/kiln_release.yml pushes. Push to kiln-release-dry for a build-only run. The shared workflow of Telosnex/telosnex_ci_tools runs the fonnx steps in tool/kiln/. tool/kiln/release.dart preserves the upstream ORT URLs when it writes the manifest. The two iOS ORT files are copied unchanged into the new immutable release.

Web

FONNX ships a local ONNX Runtime Web 1.31.1 reference runtime in lib/web (mirrored into example/web in this repository). Every model Worker imports ./ort.min.mjs; no executable CDN dependency is used. The same version backs native and Web inference, and the package manifest locks the JS module, Emscripten side module, Wasm, workers, deployed copies, and supported model fixtures. Copy the required init/Worker files plus the three ort* runtime files from lib/web into the consuming app's web/ directory.

Run the complete package gate with:

tool/test_all.sh

It downloads the models, verifies source/artifact/model hashes, executes a real local Web identity and Magika Worker model, runs the package-owned finalizer through 8,192 ASan/UBSan ownership states, checks exact native ORT artifacts plus model goldens/RSS tests serially, and runs the deterministic frontend corpus through VM, Chrome JavaScript, and Chrome Wasm. Set FONNX_FULL_RUNTIME_MATRIX=1 to include iOS Simulator, Android, Linux arm64/x64, and Windows/Wine artifact execution.

Sending these headers with the request for the ONNX JS package gives a 10x speedup:

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

See this GitHub issue for details. TL;DR: It allows use of multiple threads by ONNX's WASM implementation by using a SharedArrayBuffer.

Developing with Web

Serve .wasm as application/wasm. For multithreaded inference, serve both cross-origin isolation headers shown above; ORT falls back to one thread when SharedArrayBuffer is unavailable. flutter run -d chrome is sufficient for single-threaded development and requires no CDN download or browser extension.

License

FONNX is licensed under the MIT License. See LICENSE, which also has the licenses of the code it bundles (ONNX Runtime and ONNX Runtime Extensions).