rag_engine_flutter 0.23.0
rag_engine_flutter: ^0.23.0 copied to clipboard
Native Rust FFI plugin for mobile_rag_engine. Provides high-performance tokenization and HNSW vector indexing for on-device RAG.
Changelog #
0.23.0 #
- Used by
mobile_rag_engine0.24.0; see its changelog for what apps see. - HNSW: links are chosen with the HNSW paper's neighbour heuristic and nodes unreachable from the entry point are linked after the build, so a short document next to a dense cluster is found again. The file format is version 2; a version 1 file is rebuilt from the database on first activation. A rebuild unmaps the previously loaded file, so another collection's index is no longer searched.
- BM25: English words are stemmed (Snowball, new dependency
rust-stemmers1.2), tokens come from the NFKC form of the text (new direct dependencyunicode-normalization0.1.25), and Korean particles after Latin words, numbers and one-syllable nouns are split off. - Vector search: new
search_chunks_in_sourcesreturns cosinesimilarityfor a search limited to some sources. - Search handle: with
adjacent_chunks > 0,search_meta_hybridkeeps hits in rank order and appends their neighbours. - Sources: the duplicate check hashes the NFC text and still finds rows hashed by earlier versions; adding an already indexed document no longer marks the collection as changed; a completed source with no chunks is ingested again.
- Chunking: markdown chunks are budgeted together with their
Header Path: …line; CommonMark heading and fence rules. - Reembed:
ChunkForReembeddinggainsembedding_text, the text ingestion embeds. An interrupted reembed is detected, and a reembed can be retargeted to another model. compress_textno longer panics after a multi-byte sentence terminator and keeps3.14whole.- Engine version: the crate version now equals the package version.
0.22.0 #
- Embedding truncation: Text is cut at 512 tokens, the encoder's position limit, instead of at 256, 384 or 512 tokens chosen by character count. Text under 1,200 characters was cut at 256 tokens, so dense scripts such as Korean (about twice the tokens per character of English) lost the end of most chunks. In a desktop measurement on AutoRAGRetrieval (Korean) with a Korean e5 model, chunk-level nDCG@10 rose from 0.907 to 0.942 with semantic chunking and from 0.805 to 0.895 with markdown chunking.
- Chunking:
- The semantic chunker's token budget is 512 tokens as well, so dense text is split into fewer, fuller chunks. English text at the default chunk size is chunked as before.
- Markdown chunks, including code blocks and tables, now fit the token limit. Only plain-text sections longer than
max_charswere checked; the rest were truncated at embedding time. - Chunk overlap stays capped at 128 tokens.
- Splitting paragraphs that exceed the token budget tokenizes each paragraph once instead of once per candidate, so dense text such as Korean chunks up to 3× faster. Every chunk is still checked against the budget with an exact count.
- Existing indexes keep working, since the model and vector space are unchanged. Re-ingesting Korean, Japanese or Chinese documents is optional and recovers text that was truncated.
- HNSW: Building the vector index is about 3.8× faster (20k × 384-dim vectors: 82 s to 22 s), and vector search on a loaded index is about 2× faster. The layer search kept its results in a list that it re-sorted for every neighbor it looked at; it now uses heaps and a visited bitset, and the builder caches vector norms. Recall is unchanged.
- BM25: Korean words with attached particles or endings now match. Whole-word tokens never matched the bare stem: "검색엔진" missed "검색엔진은". Hangul words now also index their syllable bigrams. BM25 is rebuilt at activation, so existing data needs no migration.
- Embedding fingerprint:
extend_embedding_fingerprint(from, to)rewrites a stored fingerprint, a pending reembed target and the chunk tags fromfromtotoin one transaction, without touching embeddings. It only accepts atothat appends|-separated components tofrom, so it cannot be used to switch models. The Dart package uses it to add a model-content hash to fingerprints written by earlier releases.sample_embeddings_for_fingerprint(fingerprint, limit)returns the stored f32 embeddings of the shortest chunks tagged with a fingerprint, with the text they were embedded from. The Dart package re-embeds these to check that the loaded model wrote the database before it extends a fingerprint. - Empty collections:
activate_collection_for_hybrid_searchno longer fails with "HNSW cache parity failed" when a collection has no completed chunks. Rebuilding an empty collection leaves no HNSW index loaded, and the parity check expected an index with zero nodes. An empty collection that is already active is not rebuilt again on later activations, so the HNSW file left from deleted documents is not reloaded on every search. - Dependencies:
flutter_rust_bridge2.13.0, up from 2.11.1, with the bindings regenerated byflutter_rust_bridge_codegen2.13.0. Existing FFI functions are unchanged;extend_embedding_fingerprintandsample_embeddings_for_fingerprintare new.
0.21.2 #
- Search index:
- Deleted chunks no longer come back in results. SQLite reuses the ids of deleted chunks, and an index was reused whenever its chunk count matched the database. Each BM25/HNSW index now records the collection generation it was built from, and is rebuilt when the data changed. Deletes also remove ids from the active BM25 index.
- Chunks added since the last rebuild are found by BM25 after the next activation, without an explicit rebuild.
collection_index_stategains columns through additive migrations. The first activation after upgrading rebuilds the HNSW index once.- Vector search no longer writes to the database, so it doesn't wait on a running ingestion.
- DOCX extraction (replaces
docx-lite):- Spaces, tabs and line breaks are kept. Words were merged ("Hello world" became "Helloworld").
- Tables, nested tables and text boxes stay where they are in the document, with empty cells kept. Headers, footers, footnotes and endnotes follow the body. Deleted revision text is skipped.
- Decompression is limited to 100× the file size (16 MB to 200 MB), so a small crafted file can't exhaust memory.
- PDF extraction: PDFs that only restrict copying or printing (empty user password) are now read instead of rejected as encrypted. AES-encrypted PDFs are still unsupported.
- Chunking:
- Short lines inside a paragraph are no longer treated as titles. Hard-wrapped text (for example from PDFs) was split into one chunk per line.
- Every chunk, including its overlap, now fits the embedding model's token limit. The end of a chunk was silently truncated at embedding time.
overlap_charsis clamped below half ofmax_chars.
- Database pool:
close_db_poolreturns only after every pooled SQLite handle is closed. A replacement connection that the pool opened in the background could close later, so writing to or deleting the database right after closing failed withdatabase is lockedor a file-in-use error, most often on Windows. Each pool now uses a single background worker.
0.21.1 #
- Release builds: Debug builds were unaffected; these fix apps built for release.
- Android: ships a consumer ProGuard rule that keeps
ai.onnxruntime.**. R8 removed classes that ONNX Runtime's JNI code looks up by name, so release APKs aborted withClassNotFoundException: ai.onnxruntime.TensorInfoon the first embedding. - iOS: the Runner target links with
-Wl,-export_dynamic. Release builds dead-stripped the Rust functions Dart looks up at runtime (Failed to lookup symbol 'frb_get_rust_content_hash'). - iOS and macOS: the Runner target uses
STRIP_STYLE = non-global, so archives (App Store, TestFlight, Ad Hoc) keep those symbols.
- Android: ships a consumer ProGuard rule that keeps
- Store submission:
- Android: the native library is linked with 16 KB page alignment (
-z max-page-size=16384). Google Play requires this for apps targeting Android 15+; 0.21.0's library was 4 KB aligned. - iOS: ships a privacy manifest (
PrivacyInfo.xcprivacy) for the required-reason APIs the bundled SQLite uses: file timestamps (C617.1) and disk space (E174.1). App Store review rejects undeclared uses (ITMS-91053). - The new
build.rschanges the crate hash, so 0.21.1 ships newly built and signed pre-compiled binaries.
- Android: the native library is linked with 16 KB page alignment (
0.21.0 #
- Windows (x64): Declares the Windows plugin platform and publishes
windows/again. A signed pre-compiled binary forx86_64-pc-windows-msvcis built alongside the other targets. - Fix:
save_hnsw_indexunmaps the previously loaded index before renaming the new file over it. Windows refused to replace a mapped file ("Access is denied").
0.20.0 #
- Pre-compiled binaries:
- Signed binaries are published for Android (arm64-v8a, armeabi-v7a, x86_64, x86), iOS (device and simulator) and macOS (arm64, x86_64). Without
rustupon the build machine, Cargokit downloads and verifies them, so no Rust toolchain is needed. Withrustupinstalled, it still builds from source. Earlier versions never shipped binaries. - Windows and Linux are removed from the declared plugin platforms. They were never built or supported.
- Signed binaries are published for Android (arm64-v8a, armeabi-v7a, x86_64, x86), iOS (device and simulator) and macOS (arm64, x86_64). Without
- Correctness:
- Re-embedding now rewrites the quantized mmap copy of each vector instead of leaving the stale one that exact scans and HNSW read.
- Embedding dimensions that are not a multiple of 32 no longer score 0.0: the shorter final Q8_0 block is accepted.
- A truncated or corrupt
.hnswfile is rejected at load instead of panicking during search. A zero-length.vecfile is re-initialized instead of failing every launch. close_db_poolalso drops the mmap vector store, so no mapping outlives a deleted database.- Forward-ported the 0.19.3 fix: sources left in
processingby a dead process are markedfailedon the firstinit_source_dbof that database in a new process.
- Performance:
- Exact scans encode the query once per scan instead of once per row.
tokenize()no longer deep-clones the tokenizer on every call. One truncating copy is cached per max-length bucket.
- Privacy:
info!logs no longer include source names ormetadata_likepatterns.
0.20.0-dev.11 #
- Compatibility: Align the native companion with the committed Flutter Rust Bridge 2.11.1 bindings, including the required content-hash ABI symbol.
0.20.0-dev.5 #
- Bug Fix: Fixed a
PanicExceptionduring Phase 2 of HNSW search by adding a missing cursor offset skip forblob_len(vector data) before readingnode_max_layer.
0.20.0-dev.4 #
- Bug Fix: Fixed a binary header parsing bug in
custom_hnsw.rswhere the cursor size was14instead of18bytes, causingfailed to fill whole buffererrors when loading the HNSW index.
0.20.0-dev.3 #
- Critical Bug Fix: Fixed HNSW and Linear search to read quantized blobs from
MMAP_STOREviammap_idwhen SQLiteembedding_i8is empty. Fixed a false-positive in linear search fallback where empty arrays caused fake 0.0 similarity results.
0.20.0-dev.2 #
- Bug Fix: Fixed a critical bug in HNSW index rebuilding where
f32vectors were read as empty instead of being dequantized fromi8compressed blobs, causing a fallback to linear scan.
0.20.0-dev.1 #
- VABQ Quantization:
- Implemented core VABQ (Variance-aware Adaptive Block Quantization) kernel in Rust (
vector_quant.rs). - Asymmetric precision (INT8 for high-variance blocks, INT4 for low-variance blocks) optimizing for memory while preserving Recall.
- Implemented core VABQ (Variance-aware Adaptive Block Quantization) kernel in Rust (
0.19.3 #
- Ingestion recovery:
- Sources left in
processingby a process that died mid-ingest are markedfailedon the firstinit_source_dbof that database in a new process. Previously they were reported as "already in progress" forever and could not be re-ingested. Re-adding the same content now resumes ingestion. Laterinit_source_dbcalls in the same process do not touch live claims.
- Sources left in
- Tests:
- The lib test suite now compiles and passes with the shipped
vector_faer,vector_quant_i8features.
- The lib test suite now compiles and passes with the shipped
0.19.2 #
- Block-wise Quantization:
- Implemented block-wise scalar quantization (Q8_0 style) with 32-dimension block sizes.
- Implemented backwards-compatible exact-scan similarity kernel supporting dynamic fallback for legacy 768-byte uniform blobs and new 864-byte packed blobs.
- HNSW Reconstruction:
- Re-routed HNSW rebuilding to load original high-precision f32 embeddings directly, resolving the compound distortion loop and restoring semantic recall.
0.18.4 #
- PDF extraction UX:
- Preserves the
scanned/image-onlymarker for mixed scanned PDFs that also contain page-level extraction failures so host apps can route them to OCR. - Omits the OCR marker when every page fails to extract, avoiding misleading OCR guidance for fully corrupt/unsupported PDFs.
- Preserves the
- Release-path validation:
- Added i8 cosine, recall@10 floor, and cosine-fidelity safety nets for the shipped
vector_quant_i8hot path. - Added fail-closed CI guards for the shipped
vector_faer,vector_quant_i8feature combo. - Added bench-only i8 wrappers and measured i8 scan/hot-kernel baselines without changing production kernels.
- Added i8 cosine, recall@10 floor, and cosine-fidelity safety nets for the shipped
- Diagnostics:
- Logs corrupt f32 embedding blobs with row ids during index rebuild instead of silently skipping them.
0.18.3 #
- PDF extraction quality:
- Preserved paragraph boundaries while normalizing PDF text so downstream chunking can split on semantic breaks instead of one long flattened body.
- Added per-page PDF extraction with per-page fallback so a single malformed page no longer aborts the whole document.
- Deduplicated dense adjacent double-rendered text artifacts seen in Korean PDFs while guarding natural repetitions and English words from false-positive rewrites.
- Surfaced scanned/image-only or effectively empty PDF extraction as an error instead of silently indexing an unsearchable 0-chunk source.
0.18.2 #
- Scoped BM25 search:
- Added scoped BM25 ranking over the existing in-memory inverted index, restricted by the scoped chunk ids collected during exact vector scan.
- Removed query-time chunk-body reads/tokenization from the source/metadata scoped exact-scan path when BM25 is enabled.
0.18.1 #
- Ingest session:
- Added extracted body byte/character lengths to
PreparedIngestionso Dart file-ingest callers can display body-size metadata without materializing the full body. - Kept
prepare_source_ingestion_from_fileon the path-only fast path: the extracted body is measured in Rust while document text still avoids Dart FFI body transfer.
- Added extracted body byte/character lengths to
0.18.0 #
- Retrieval hot path (copy-minimized Rust core):
- Added
search_hnsw_slice(&[f32], usize)andsearch_hybrid_inner(String, &[f32], ...)as borrowed-slice variants of the existing FFI-public entrypoints. The owned-Vec<f32>signatures ofsearch_hnswandsearch_hybridare preserved forflutter_rust_bridgecompatibility and now delegate to the slice helpers. - Updated
search_meta_hybrid's retry loop to borrow the query text, embedding, and options across attempts. A transientConcurrentMutationno longer forces per-attemptVec<f32>/String/SearchMetaHybridOptionsclones. - Refactored the parallel vector + BM25 fan-out inside
search_hybrid_innerto capture the embedding by reference insidestd::thread::scoperather than cloning per spawn. - Replaced the per-file
decode_f32_embeddingcopies insource_rag.rs,hybrid_search.rs, andsimple_rag.rswith a single sharedvector_math::decode_f32_embeddingso the SQLite-BLOB →Vec<f32>decode path is owned in one place.
- Added
- Quantization storage path (
vector_quant_i8feature):- Added
quantize_f32_to_u8_blob(&[f32]) -> (Vec<u8>, f32)which produces the SQLite BLOB representation directly without the intermediateVec<i8>thatquantize_f32_to_i8+i8_blob_from_slicepreviously required. The two paths are byte-for-byte identical and covered by a parity regression test. - Switched the four ingest/migration callsites (
source_rag.rs,simple_rag.rs) that store quantized embeddings to the direct blob path.
- Added
0.17.0 #
- Low-level lane:
- Added a generation-pinned
SearchHandlewith metadata-first hybrid search, batch hydration, excerpt fetch, and Rust-side context assembly. - Added bounded
SearchHitMetaplusStaleSearchHandle/ConcurrentMutationerror paths for low-level handle operations.
- Added a generation-pinned
- Search / Context:
- Added Rust-side context assembly primitives that preserve the current header-path contextual injection and exact engine-tokenizer budgeting contract.
- Tightened
SearchHandlehydration/excerpt/assembly generation checks to reject stale handles and mid-operation mutations deterministically.
- Ingest:
- Added UTF-8 bytes and file-based extraction fast paths for additive low-level ingest APIs.
- Test:
- Added regression coverage for generation bumps, stale-handle rejection, concurrent mutations during hydration/assembly, duplicate hydration requests, and low-level lane parity.
0.16.0 #
- Chunking:
- Hardened plain-text heading detection against short imperative, UI-label, and command-style false positives.
- Clarified the markdown table split contract around raw source-backed slices and non-repeated later headers.
- Parser / Quality:
- Added parser regression coverage for mixed Unicode/PDF-like artifacts.
- Added an ignored
join_pages()stress benchmark for release-time performance checks.
0.15.0 #
- Feat:
- Added non-truncating
count_tokenssupport for exact engine-tokenizer budgeting paths.
- Added non-truncating
- Chunking:
- Tightened plain-text chunk planning with heading-aware boundaries and boundary-snapped overlap.
- Standardized chunk offsets around raw source byte ranges for both plain-text and markdown paths.
- Parser / Runtime:
- Promoted CJK newline restoration into the production page-join path.
- Removed experimental CJK helper leftovers and reduced avoidable clone-heavy paths.
0.14.2 #
- Fix:
- Normalized PDF extraction artifacts (private-use, noncharacter, and control code points) into safe separators to prevent broken box characters in extracted text.
- Applied normalization before page-number trimming and dehyphenation to improve downstream chunking and embedding stability.
- Test:
- Added regression coverage for malformed PDF spacing artifacts.
0.14.1 #
- Feat:
- Added ingestion safety with claim/status/clear APIs and status-aware indexing in vectors.
- Refactor:
- Refactored
vector_mathmodule to tighten internal function visibility.
- Refactored
0.14.0 #
- Vector Math Refactor:
- Removed
ndarraydependency; introducedvector_mathmodule with allocation-free dot product, L2 norm, and cosine kernels. - Added optional
vector_faerfeature usingfaercrate for SIMD-accelerated vector operations.
- Removed
- i8 Scalar Quantization (
vector_quant_i8feature):- Added
vector_quantmodule withquantize_f32_to_i8,dequantize_i8_to_f32, and i8 cosine similarity kernels. - Updated
init_db/init_source_dbschema migration to addembedding_i8andembedding_scalecolumns. - Linear scan search paths (
search_similar,search_chunks_linear_in_collection) use quantized cosine when available, with f32 fallback.
- Added
- Benchmark API:
- Added
benchmark_search_linear_scan()andbenchmark_search_chunks_linear_in_collection()FFI entrypoints for deterministic linear scan measurement.
- Added
- Code Quality:
- Applied
rustfmtformatting across all source files. - Sorted module declarations alphabetically in
mod.rs. - Removed unused
ndarraycrate dependency fromCargo.toml.
- Applied
0.13.0 #
- Multi-Collection Core:
- Added collection-scoped schema and lifecycle support (
collection_idon sources/chunks, collection index state). - Added collection-aware source/chunk APIs for list/add/delete/stats/rebuild/search paths.
- Preserved legacy compatibility through default collection mapping.
- Added collection-scoped schema and lifecycle support (
- Hybrid Search Isolation:
- Extended
SearchFilterwithcollection_id. - Updated exact-scan switching rules to trigger on source/metadata filters while preserving collection post-filter behavior.
- Added collection-scoped activation hook to align BM25/HNSW in-memory indexes before hybrid search.
- Extended
- Reliability & Recovery:
- Improved collection activation and index-state transitions for load/rebuild flows.
- Added/expanded tests for collection isolation, scoped dedupe, and filter semantics.
0.12.0 #
- Logger Stability (Hot Restart):
- Reworked Dart log sink ownership to
Arc<StreamSink<_>>for safer cross-thread access. - Avoided holding logger locks while sending logs to Dart stream.
- Switched log stream teardown to non-blocking cleanup to prevent restart deadlocks.
- Added stale sink recovery on stream send failures.
- Reworked Dart log sink ownership to
0.11.0 #
- Hybrid Search:
- Improved source-filter exact-scan path to keep scoped BM25 ranking.
- Added regression test for source-filter + exact-keyword behavior.
- Tokenizer:
- Added dynamic truncation policy by input length (256/384/512).
- Chunking:
- Applied overlap prefix logic in
semantic_chunk_with_overlap.
- Applied overlap prefix logic in
- BM25:
- Improved tokenization to retain meaningful single-char CJK/code tokens.
0.10.2 #
- Fix: Corrected HNSW index loading path resolution in
load_hnsw_index. - Fix: Filtered out verbose debug logs from
hnsw_rscrate. - Stabiity: Handled uninitialized/empty index cases in
save_hnsw_indexto prevent crashes.
0.10.1 #
- Maintenance release:
- Fix hnsw uninitialized error.(caused by updating hnsw cargo version)
0.10.0 #
- Maintenance release:
- Updated dependencies.
- Internal improvements for
mobile_rag_enginecompatibility.
0.9.1 #
- Improved markdown chunking logic:
- Structure Preservation: Code blocks and tables are now split intelligently, preserving their type (
code,table) instead of reverting to plain text. - Code Block Linking: Large code blocks split into multiple chunks now carry metadata (
batch_id,batch_index,batch_total) to allow reconstructing the original block. - Table Header Repetition: Large tables split across chunks now automatically repeat the header row in every chunk to maintain column context.
- Structure Preservation: Code blocks and tables are now split intelligently, preserving their type (
0.9.0 #
- Thread Configuration: Added support for explicit thread count configuration in ONNX Runtime.
- Memory Optimization: Model loading now supports direct file path usage to reduce memory overhead.
- Dependencies: Bumps
mobile_rag_enginecompatibility.
0.8.0 #
- Exact Scan Optimization: Implemented brute-force vector scan for source-filtered searches, guaranteeing perfect recall within the selected source.
- Smart Dehyphenation: Fixed Korean text extraction to correctly handle words split by line breaks.
- Dependencies: Bumps
mobile_rag_enginecompatibility.
0.7.6 #
- Duplicate Logs Fix: Logger now only uses
println!when Dart stream is not connected, preventing duplicate output. - Log Format: Simplified log format to
[LEVEL] message(removed redundant tags).
0.7.5 #
- BM25 Index Fix: Added
rebuild_chunk_bm25_index()function to properly build BM25 index for Source RAG chunks. - Hybrid Search Fix: BM25 keyword search now correctly works alongside Vector similarity search.
- Initialization: Both HNSW and BM25 indexes are now built during app initialization for existing chunks.
0.7.0 #
- Metadata Support: Added
metadatacolumn tosourcestable and support inHybridSearchResult. - Hybrid Search: Enhanced
search_hybridwith weighted scoring (Vector + BM25) and metadata retrieval. - Prompt Optimization: Search results now include metadata for better LLM context construction.
0.6.1 #
- Updated README to remove specific version constraints in examples.
- Updated Supported Platforms documentation.
0.6.0 #
- DB Connection Pool: Implemented connection pooling with
r2d2for 50-90% search performance improvement - Resource Optimization: Eliminated redundant SQLite connections to reduce file descriptor usage
- Refactoring: Updated API to use pooled connections instead of direct file opens
0.5.1 #
- Unit Tests: Added tests for
hnsw_indexanddocument_parsermodules - BM25 Korean Support: Improved Korean tokenization using
unicode-segmentationcrate for better word boundary detection - Code Quality: Enhanced test coverage for core Rust modules
0.5.0 #
- PDF/DOCX Text Extraction: New text extraction with smart dehyphenation
- Markdown Chunking: Structure-aware chunking with header path inheritance
- PDF Fix: Enhanced text normalization to preserve paragraph structure
- Safety: Added 50MB file size processing limit
0.4.0 #
- Fix binary mismatch: Rebuilt native binaries to resolve hash mismatch with Dart bindings.
0.3.0 #
- Fix platform directories missing: Include ios/, android/, macos/, linux/, windows/ in package
- Add .pubignore to prevent parent ignore rules from excluding platform configs
0.2.0 #
- Fix package structure: Include rust/ directory in package for correct pub.dev distribution
- Update platform build configs (iOS, macOS, Android, Linux, Windows) to reference internal rust/ path
0.1.0 #
- Initial release
- High-performance tokenization with HuggingFace tokenizers
- HNSW vector indexing for O(log n) similarity search
- SQLite integration for persistent vector storage
- Semantic text chunking with Unicode boundary detection
- Prebuilt binaries for iOS, macOS, and Android