mobile_rag_engine 0.22.2
mobile_rag_engine: ^0.22.2 copied to clipboard
Build local/on-device RAG in Flutter with Dart APIs, offline semantic search, HNSW+BM25 hybrid retrieval, and text-layer PDF indexing.
Mobile RAG Engine #
Build local, on-device RAG in Flutter with a Dart package.
Mobile RAG Engine is a Flutter package for local Retrieval-Augmented Generation (RAG): ingest local documents, chunk and embed them on-device, then run hybrid semantic + keyword search through a Dart API. No server, no API cost, no network round-trip for retrieval.
Use it when you need a Flutter local RAG engine for private notes, document Q&A, chat with PDF, offline assistants, or enterprise apps where user data must stay on the device.
Retrieval only. This package finds and assembles the relevant context. Generating an answer with an LLM is your app's separate step.
This README describes 0.22.2. Still on 0.20.x? It has no Model Pack
setupcommand orRagModelPack, and it always builds the Rust core from source; follow thev0.20.1README instead.
Why this package? #
No Rust Toolchain Required (0.21.0+) #
Starting with 0.21.0, the Rust core (rag_engine_flutter) ships as signed,
pre-compiled binaries for Android (arm64-v8a, armeabi-v7a, x86_64, x86),
iOS (device and simulator), macOS (arm64, x86_64), and, from 0.22.0,
Windows (x64).
Cargokit downloads the binary for your
target during flutter build, verifies its signature, and links it, so you do
not need Rust, Cargo, or rustup.
- The first build of each platform downloads the binary from this repository's
GitHub releases, so the build machine needs access to
github.com. - If
rustupis installed, Cargokit builds the Rust core from source instead. That path needs Rust stable, and its first build takes longer. - A binary whose signature does not verify is ignored rather than linked.
Performance #
| Feature | Pure Dart | Mobile RAG Engine (Rust) |
|---|---|---|
| Tokenization | Slow | HuggingFace tokenizers (Rust) |
| Vector Search | O(n) | HNSW Index — sub-linear retrieval |
| Memory Usage | High | Copy-minimized Rust core, Float32List zero-copy transport |
Numbers vary by device and corpus. See
benchmark_serviceand the0.18.0retrieval-hot-path notes in CHANGELOG.md for measured deltas on your own hardware.
Supported and Verified Scope #
| Area | Current status | Evidence / boundary |
|---|---|---|
| Local Flutter RAG retrieval | Supported | Dart facade over a Rust core for ingest, chunking, embedding, SQLite storage, HNSW vector search, BM25 keyword search, RRF fusion, and context assembly |
| Offline / on-device operation | Supported | Models and user documents stay local after you bundle the ONNX model and tokenizer assets |
| Hybrid source retrieval | Verified on benchmark fixtures | 80-source balanced profile run: source_recall@10 = 1.000 for shipped default_hybrid |
| Passage/context retrieval | Verified on benchmark fixtures | 80-query passage run: passage_recall@10 = 0.925, answerable_context@10 = 0.938; semantic passage misses remain the main improvement area |
| Text-layer PDF-to-RAG | Verified on sample scope | sample_eng.pdf and sample_kor.pdf profile run: 8/8 PDF-derived queries reached source, passage, and answerable context at top-10 |
| Scanned/image-only PDFs | Detected, not OCR-processed | OCR-required PDFs are surfaced as extraction errors so your app can route to an OCR layer; OCR is not bundled in this package |
| Large, table-heavy, OCR-heavy PDFs | Still being validated | Do not treat the PDF smoke result as broad PDF robustness or mobile latency/memory proof |
For the implementation-oriented guide, see Flutter Local RAG Engine Guide.
100% Offline & Private #
Data never leaves the user's device. Perfect for privacy-focused apps (journals, secure chats, enterprise tools).
Features #
End-to-End RAG Pipeline #
One package, complete pipeline. From any document format to LLM-ready context.
Key Features #
| Category | Features |
|---|---|
| Document Input | Text-layer PDF, Markdown, Plain Text, and beta DOCX support; file-path and UTF-8 ingest fast paths |
| Chunking | Plain-text paragraph/line chunking with heading-aware split and tokenizer hard guard; Markdown structure-aware chunking with header-path metadata |
| Search | HNSW vector + BM25 keyword hybrid search with RRF fusion; metadata-first search with explicit context/chunk hydration |
| Storage | SQLite persistence, HNSW Index persistence (fast startup), connection pooling, resumable indexing |
| Collections | Collection-scoped ingest/search/rebuild via inCollection('id') |
| Performance | Rust core, HuggingFace tokenizers, thread control, memory optimized |
| Context | Engine-tokenizer exact context budget, adjacent chunk expansion, single source mode |
Support boundaries: text-layer PDFs are production-ready. Scanned or image-only PDFs should be routed through an OCR layer before indexing. DOCX extraction is available for early adopters, but complex DOCX layouts such as tables, headers, and footnotes should be treated as beta.
Requirements #
| Platform | Minimum Version |
|---|---|
| iOS | 16.0+ |
| Android | API 21+ (Android 5.0 Lollipop) |
| macOS | 14.0+ |
| Windows | Windows 10+ (x64), from 0.22.0 |
| Build host | No Rust needed (see No Rust Toolchain Required); network access to github.com on the first build |
Windows on ARM64 and Linux are not supported. Windows builds need Visual Studio
with the "Desktop development with C++" workload, as for any Flutter Windows
app. On Windows, the engine stores its database, index files, and copied model
assets in the app support directory (%APPDATA%\<company>\<app>) instead of the
user's (often OneDrive-synced) Documents folder; other platforms are unchanged.
ONNX Runtime is provided through
flutter_onnxruntime. CocoaPods builds on both iOS and macOS need static framework linkage (use_frameworks! :linkage => :static) and deployment targets of iOS 16.0+ and macOS 14.0+, whileflutter creategenerates iOS 13.0 / macOS 10.15 with dynamic frameworks. From 0.22.1, thesetupcommand below applies these settings for you; on 0.22.0, set them by hand as shown in the Quick Start. Android release builds need ONNX Runtime classes kept from R8; from 0.22.2 (rag_engine_flutter0.21.1) the package ships that rule, the iOS/macOS release linker settings, 16 KB Android page alignment and an iOS privacy manifest (see Troubleshooting for earlier versions).
Installation #
1. Add the dependency #
dependencies:
mobile_rag_engine: ^0.22.2
2. Install the verified Model Pack #
dart run mobile_rag_engine:setup --preset stable-minilm-l6-v2-arm64-en
dart run mobile_rag_engine:setup --preset stable-minilm-l6-v2-arm64-en --check
It writes assets/mobile_rag/model.onnx, tokenizer.json, and
model-pack.json. From 0.22.1 it also raises the iOS and macOS host settings
the native dependencies need (Podfile platform, static framework linkage and the
Runner deployment target) and prints each change; existing higher targets are
kept, and --skip-host-config turns this off. Add the directory to your app
assets:
flutter:
assets:
- assets/mobile_rag/
Then initialize with one manifest:
await MobileRag.initialize(
modelPack: const RagModelPack.asset(
'assets/mobile_rag/model-pack.json',
),
);
The first success is add documents, rebuild the index, and get non-empty chunks/context from search. LLM generation remains your app's separate step.
The Model Pack preset is English-only (all-MiniLM-L6-v2). For Korean, CJK,
or other multilingual documents, use a multilingual ONNX model through the
original two-asset initializer, which remains available as an advanced path;
see the
Model Setup Guide.
Quick Index #
Features #
- Adjacent Chunk Retrieval - Fetch surrounding context.
- Index Management - Stats, persistence, and recovery.
- Markdown Chunker - Structure-aware text splitting.
- Multi-Collection - Isolate ingest/search/rebuild by category.
- Prompt Compression - Reduce token usage.
- Search by Source - Filter results by document.
- Search Strategies - Tune ranking and retrieval.
Guides #
- Flutter Local RAG Engine Guide - Build local/on-device RAG in Flutter with Dart APIs.
- Quick Start - Setup in 5 minutes.
- Model Setup - Choosing and downloading models.
- Release Build - Bundle size optimization for production.
- Troubleshooting - Common fixes.
- FAQ - Frequently asked questions.
Testing #
- Unit Testing - Mocking for isolated tests.
Usage #
Minimal Setup #
Initialize the engine once in your main() function. See the Quick Start Guide for the full parameter table.
await MobileRag.initialize(
modelPack: const RagModelPack.asset(
'assets/mobile_rag/model-pack.json',
),
deferIndexWarmup: true,
);
// Before first search, wait for BM25/HNSW warmup if you deferred it:
if (!MobileRag.instance.isIndexReady) {
await MobileRag.instance.warmupFuture;
}
Adding Documents and Searching #
class MySearchScreen extends StatelessWidget {
Future<void> _search() async {
// 1. Add Documents (auto-chunked & embedded). Indexing is auto-managed
// (debounced ~500ms) — only call rebuildIndex() if you need it now.
await MobileRag.instance.addDocument(
'Flutter is a UI toolkit for building apps.',
);
// File / UTF-8 fast paths are useful for large local documents.
await MobileRag.instance.addDocumentFromFile('/path/to/manual.pdf');
final noteBytes = await File('/path/to/notes.md').readAsBytes();
await MobileRag.instance.addDocumentUtf8(noteBytes, name: 'notes.md');
// 2. Search with LLM-ready context
final result = await MobileRag.instance.search(
'What is Flutter?',
tokenBudget: 2000,
);
print(result.context.text); // Ready to send to LLM
}
}
Handling File Picker Fallback #
addDocumentFromFile is the fastest path because the Rust core reads and chunks the file directly. Some platform pickers (cloud-backed pickers, content URIs without a stable local path, etc.) return data that is not exposed as a real filesystem path. In those cases, fall back to UTF-8 or parsed-text ingestion:
try {
await MobileRag.instance.addDocumentFromFile(path, name: fileName);
} on RagError {
final bytes = await File(path).readAsBytes();
final lower = fileName.toLowerCase();
if (lower.endsWith('.txt') ||
lower.endsWith('.md') ||
lower.endsWith('.markdown')) {
await MobileRag.instance.addDocumentUtf8(bytes, name: fileName);
} else {
try {
final text = await DocumentParser.parse(bytes);
await MobileRag.instance.addDocument(text, name: fileName);
} catch (error) {
if (DocumentParser.isOcrRequiredPdfExtractionError(error)) {
throw UnsupportedError(
DocumentParser.userMessageForExtractionError(error),
);
}
rethrow;
}
}
}
Metadata-First Search #
Use searchMeta when you want lightweight search metadata first, then explicitly assemble context or hydrate only the chunks you need.
final meta = await MobileRag.instance.searchMeta(
'What is Flutter?',
topK: 10,
);
try {
final context = await MobileRag.instance.assembleContext(
searchHandle: meta.handle,
tokenBudget: 2000,
);
final chunkIds = meta.hits.map((hit) => hit.chunkId.toInt()).toList();
final chunks = await MobileRag.instance.hydrateChunks(
searchHandle: meta.handle,
chunkIds: chunkIds,
);
final excerpts = await MobileRag.instance.getChunkExcerpts(
searchHandle: meta.handle,
chunkIds: chunkIds,
maxBytes: 256,
);
print(context.text);
print('hydrated=${chunks.length}, excerpts=${excerpts.length}');
} finally {
await meta.handle.dispose();
}
Multi-Collection (v1) #
Use collection scopes when you want independent rebuild boundaries per category.
final business = MobileRag.instance.inCollection('business');
final travel = MobileRag.instance.inCollection('travel');
await business.addDocument('Quarterly planning memo...');
await travel.addDocument('Kyoto itinerary...');
if (!travel.isIndexReady) {
await travel.warmupFuture;
}
final travelHits = await travel.searchHybrid('itinerary');
print(travelHits.length);
If you do not specify a collection, the engine uses the default __default__
collection for backward compatibility.
Advanced Usage: For fine-grained control, use the high-level metadata lane (
searchMeta,assembleContext,hydrateChunks,getChunkExcerpts) and the public API reference. Most apps should stay on theMobileRagfacade.
Sample App #
The example/
app in this repository exercises ingest, chunking, and search against the
current source.
mobile-ondevice-rag-desktop
is an older desktop demo that pairs retrieval with LLM generation (Gemma 2B
through a local Ollama server). It is built against
mobile_rag_engine 0.14.x and predates the Model Pack setup, so treat it as a
reference for wiring retrieval into an LLM prompt, not as a copy-paste starting
point.
Contributing #
Bug reports, feature requests, and PRs are all welcome!
License #
This project is licensed under the MIT License.