phoenixdb 4.1.1

ACID-compliant embedded database for Dart and Flutter, written in Rust with dart:ffi: MVCC key/value storage, SQL, HNSW vector search, hybrid document search and RAG.

PhoenixDB

An ACID-compliant embedded database engine written in Rust and exposed to Dart/Flutter through a zero-overhead dart:ffi layer. Use it as a key/value store, as typed preferences, through a small SQL front end, or as an on-device vector database with hybrid search, RAG and LLM clients.

Features

Pillar	Implementation
Index	B+Tree, 4 KiB slotted pages, configurable fill factors (min 50%, max 100%)
Integrity	CRC32 on every page, verified on every read; corruption returns Err, never bad data
Persistence	mmap for zero-copy reads + write-ahead log with fsync/sync_all at commit
Concurrency	MVCC snapshot isolation — many concurrent readers, single writer via parking_lot::RwLock
Transactions	Begin / Insert / Delete / Commit / Rollback WAL records with crash recovery
SQL	Optional: CREATE/INSERT/SELECT/UPDATE/DELETE, bound parameters, enforced primary keys, aggregates with GROUP BY, ORDER BY/LIMIT/OFFSET, caller transactions
Vector search	HNSW k-NN over memory-mapped f32 vectors; cosine / L2 / dot product; runtime-dispatched AVX2+FMA and NEON kernels; filtered and batch search
Document collections	Text + JSON metadata + embeddings in one store: hybrid vector/BM25 search, MongoDB-style metadata filters served from an index, RRF or weighted fusion, MMR diversification, newest-first paging, verify/compact/backup
Reactive queries	Watch committed changes to a key prefix or a collection's documents as a Stream; bounded queues report what a slow consumer missed
Recovery	verify and check report damage; salvage recovers every readable pair from a file too broken to open
AI toolkit	package:phoenixdb/ai.dart: Claude and OpenAI-compatible chat clients with streaming and tool calling, an agent loop, embedders, chunking, RAG with citations, semantic cache, conversation memory
FFI	Pure C ABI (extern "C"), header auto-generated by cbindgen
Dart	NativeFinalizer for automatic cleanup, Uint8List keys/values, isolate-based async API

Available as libraries, not yet in the engine's path

These modules are implemented and tested, but the storage engine does not route through them yet. They are usable directly from Rust; they do not change how PhoenixDatabase behaves today.

Module	State
lsm	MemTable, block-based SSTables with Bloom filters, leveled compaction, crash-safe durable manifest. Database still writes through the B+Tree only.
security::encryption	AES-256-GCM page encryption (encrypt-then-checksum). Not wired into the pager.
security::rbac, security::audit	Role checks with constant-time comparison; append-only audit log. Not enforced at the FFI boundary.

Key and value length validation from security is enforced on every FFI call.

Feature flags

The default build stays small for Flutter — it pulls in no heavy dependencies. Opt into what you need:

phoenixdb = { version = "4.1", features = ["sql", "encryption"] }

Flag	Adds
sql	SQL lexer, parser, and executor
encryption	AES-256-GCM encryption at rest
json	No-op, kept for compatibility (JSON support is always compiled in)
metrics	Prometheus exposition of the metrics registry
full	Everything above

Architecture

  Dart (dart:ffi)  ->  ffi.rs (C ABI, validation)  ->  Database
                                                          |
                                       +------------------+------------------+
                                       |                  |                  |
                                    txn.rs             btree.rs            wal.rs
                                   (MVCC)             (index)           (durability)
                                       |                  |
                                       +------ pager.rs --+  (cache + CRC + mmap)

Page layout (4096 bytes)

Offset	Size	Field
0	4	crc32 — checksum of bytes [4..4096]
4	4	page_id
8	4	parent_id
12	4	extra (next leaf / leftmost child / next overflow)
16	2	num_keys
18	2	cell_start
20	1	page_type
21	1	is_leaf
22	2	flags
24	8	lsn

A slot directory grows upward from byte 32 while variable-length cells grow downward from byte 4096.

Platform support

Platform	Supported	Prebuilt binary ships	Rust needed to build your app
Android — arm64-v8a, armeabi-v7a, x86_64, x86	✅	✅	No
iOS — device + simulator	✅	❌	Yes (+ Xcode)
macOS — Apple silicon + Intel	✅	❌	Yes (+ Xcode)
Linux — x86_64, aarch64	✅	❌	Yes
Windows — x86_64	✅	❌	Yes
Web	❌	—	—

Web is not supported and cannot be. The engine is native code reached over dart:ffi, which does not exist in a browser, and the browser has no filesystem for a durable page store. A web build fails to compile rather than degrading at runtime.

Android is the one platform that needs nothing extra: the NDK cross-compiles cleanly from any host, so the four ABIs ship prebuilt and the Gradle plugin packages them into your APK/AAB. Every other platform compiles the engine from source during your app build — Apple platforms because linking needs an Xcode SDK and code signing that only exist on a developer's Mac, desktop because the prebuilt libraries in native/ target the host rather than your app bundle.

Install the toolchain once on any machine that builds for iOS, macOS, Linux, or Windows:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh   # or rustup.rs on Windows

For plain Dart (no Flutter), the prebuilt libraries in native/<triple>/ are used directly and no Rust toolchain is required — see Plain Dart.

Installation

flutter pub add phoenixdb

Or add it to pubspec.yaml:

dependencies:
  phoenixdb: ^4.1.1

Use flutter pub, not dart pub. PhoenixDB is a Flutter FFI plugin, so its pubspec declares flutter.plugin.platforms. pub only accepts that key alongside a flutter: SDK bound, and that bound makes standalone dart pub refuse the package:

Because phoenixdb requires the Flutter SDK, version solving failed.

This is a packaging rule, not a code dependency — nothing under lib/ imports package:flutter. Flutter bundles a Dart SDK, so a pure Dart CLI or server works fine: resolve with flutter pub get, then run it with dart run as usual.

How the native library is delivered

PhoenixDB is an FFI plugin, so the engine ships as a native artifact rather than Dart source. What happens per platform:

Platform	Delivery	Built when
Android	Prebuilt .so in android/src/main/jniLibs/<abi>/, packaged into your APK/AAB	Shipped prebuilt
iOS	Static archive compiled by rust/build-apple.sh, linked into the app binary	flutter build ios
macOS	Universal static archive (arm64 + x86_64)	flutter build macos
Linux	cargo build driven by CMake, bundled next to the executable	flutter build linux
Windows	cargo build driven by CMake, DLL bundled with the app	flutter build windows

Android ships prebuilt because the NDK cross-compiles cleanly from any host. Apple platforms build from source at app-build time because linking requires an Xcode SDK and code signing that only exist on the developer's Mac.

Flutter apps need no extra setup — the platform folders are wired up automatically. Building for iOS/macOS/Linux/Windows additionally requires a Rust toolchain on the build machine:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Plain Dart (non-Flutter)

There is no app bundle to carry the library, so build it once and leave it where the loader can find it:

git clone https://github.com/ayoubzulfiqar/phoenixdb && cd phoenixdb
./build.sh                    # installs into native/

The loader searches native/<target-triple>/, then native/, then rust/target/{release,debug}/. You can also point at it explicitly:

final db = PhoenixDatabase.open('data.pdb', libraryPath: '/opt/lib/libphoenixdb.so');

Building

./build.sh              # release build, installs into native/
./build.sh --debug
./build.sh --all        # every supported platform (needs cargo-zigbuild)
./build.sh --target aarch64-apple-darwin

.\build.ps1
.\build.ps1 -Target x86_64-pc-windows-msvc

Cross-compilation

Cross builds need a linker that emits the target's object format. The most portable option is cargo-zigbuild, which uses Zig's bundled clang and needs no Xcode installation for macOS:

cargo install cargo-zigbuild
winget install zig.zig     # or: brew install zig / apt install zig

rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu \
                  x86_64-apple-darwin aarch64-apple-darwin
./build.sh --all

build.sh detects cargo-zigbuild and uses it automatically for any target that is not the host. Without it, the script falls back to plain cargo and honours the standard CC / AR variables:

CC=aarch64-linux-gnu-gcc AR=aarch64-linux-gnu-ar \
  ./build.sh --target aarch64-unknown-linux-gnu

Prebuilt binaries

Cross-compiled libraries are installed into native/<triple>/, and the Dart loader prefers the directory matching the running architecture before falling back to the flat native/ layout:

Target triple	Artifact	Format
x86_64-unknown-linux-gnu	libphoenixdb.so	ELF 64-bit x86-64
aarch64-unknown-linux-gnu	libphoenixdb.so	ELF 64-bit ARM aarch64
x86_64-apple-darwin	libphoenixdb.dylib	Mach-O 64-bit x86_64
aarch64-apple-darwin	libphoenixdb.dylib	Mach-O 64-bit arm64
x86_64-pc-windows-gnu	phoenixdb.dll	PE32+ x86-64

All five export the same 19-function C ABI. Binaries are build output and are not tracked in git — run ./build.sh --all to regenerate them.

The C header is regenerated into native/include/phoenixdb.h on every build.

Usage

Synchronous

import 'package:phoenixdb/phoenixdb.dart';

final db = PhoenixDatabase.open('data.pdb');
db.insert(utf8Key('hello'), utf8Value('world'));
print(utf8Decode(db.getOrThrow(utf8Key('hello'))));  // world

db.transaction((txn) {
  db.insert(utf8Key('a'), utf8Value('1'), txnId: txn);
  db.insert(utf8Key('b'), utf8Value('2'), txnId: txn);
});

db.close();

Asynchronous (recommended for Flutter)

Every call runs on a dedicated worker isolate, so disk I/O never blocks the UI:

final db = await AsyncPhoenixDB.open('data.pdb');
await db.insert(utf8Key('k'), utf8Value('v'));
final value = await db.get(utf8Key('k'));
await db.close();

Preferences

A shared_preferences-style typed facade over the same store — durable, transactional, and with no platform channel:

final prefs = await PhoenixPrefs.open('settings.pdb');

await prefs.setString('theme', 'dark');
await prefs.setInt('launches', 42);
await prefs.setBool('onboarded', true);

print(await prefs.getString('theme'));   // dark
print(await prefs.getInt('missing'));    // null

await prefs.close();

SQL

Requires a library built with the sql feature; check db.supportsSql if you target lean embedded builds.

final db = PhoenixDatabase.open('app.pdb');

db.query('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)');
db.query("INSERT INTO users VALUES (1, 'alice'), (2, 'bob')");

final result = db.query('SELECT name FROM users WHERE id > 1 ORDER BY name');
print(result.rows);           // [[bob]]
print(result.scalar);         // bob   (single cell shorthand)
print(result.asMaps);         // [{name: bob}]

db.query("UPDATE users SET name = 'carol' WHERE id = 2").affected;  // 1
db.close();

On a UI isolate, use the async form so parsing and execution stay off the main thread:

final db = await AsyncPhoenixDB.open('app.pdb');
final r = await db.query('SELECT name FROM users WHERE id = 1');
print(r.scalar);  // alice

SQL rows and raw key/value entries live in the same database without interfering — the SQL layer namespaces its keys.

Bind values with ? (or ?1, ?2, …) instead of interpolating them — a bound value is never parsed as SQL:

db.query('SELECT name FROM users WHERE id = ? AND name LIKE ?', params: [2, 'b%']);

Supported: CREATE TABLE / DROP TABLE (with IF [NOT] EXISTS), INSERT, SELECT (projection with aliases, WHERE with AND/OR/NOT, IN, BETWEEN, LIKE/ILIKE, IS [NOT] NULL; COUNT/SUM/AVG/MIN/MAX with DISTINCT and GROUP BY; ORDER BY, LIMIT, OFFSET), UPDATE, DELETE. PRIMARY KEY columns are unique and indexed, and statements can join a caller's transaction with txnId:. Not supported: joins, subqueries, HAVING, secondary indexes.

Vector search

An embedded HNSW k-NN index for local-first semantic search — no server, no second database.

final index = PhoenixVectorDB.open(
  'vectors.pvec',
  dimensions: 384,          // fixed at creation; must match on reopen
  metric: VectorMetric.cosine,
);

index.insert('doc-1', embedding);           // Float32List
index.insertAll({'doc-2': a, 'doc-3': b});

for (final match in index.search(VectorQuery(query, k: 5))) {
  print('${match.id}  ${match.score.toStringAsFixed(4)}');
}

index.save();   // syncs vectors + writes the HNSW snapshot
index.close();

On a UI isolate, use the async form — every call runs on a worker isolate, so a search cannot drop a frame:

final index = await AsyncPhoenixVectorDB.open('vectors.pvec', dimensions: 384);
final hits = await index.search(VectorQuery(query, k: 5));
await index.close();

Metric	distance	score (higher is better)
VectorMetric.cosine	1 - cos(θ), in [0, 2]	cosine similarity, [-1, 1]
VectorMetric.euclidean	true L2 distance	1 / (1 + d), in (0, 1]
VectorMetric.dotProduct	-(a·b)	the inner product

Notes:

• Vectors are the source of truth. The graph is snapshotted separately and rebuilt automatically if the snapshot is missing, stale or corrupt.
• Small indexes are exact. Below 512 live vectors the engine scans exhaustively, so a handful of documents gives exact answers, not approximate ones.
• Removal is a tombstone, so ids in the graph stay stable. Call compact() once stats().deletedRatio passes roughly 0.3.
• efSearch trades latency for recall. The default (64) suits k ≤ 10; raise it for higher recall on large, high-dimensional collections.
• SIMD is automatic. index.kernel reports avx2+fma, neon or portable. AVX2 is selected at runtime, never baked in, so a published binary still runs on pre-2013 x86 hardware.

Document collections and hybrid search

A PhoenixCollection keeps each document's text, JSON metadata and embedding together, so one call can combine semantic similarity, keyword relevance and structured filters:

final kb = await AsyncPhoenixCollection.open('kb', dimensions: 384);

await kb.upsert([
  Document('doc-1',
      text: 'PhoenixDB runs entirely on-device',
      metadata: {'lang': 'en', 'year': 2025, 'tags': ['db', 'mobile']},
      vector: embedding),
]);

final hits = await kb.search(
  vector: queryEmbedding,                 // HNSW nearest neighbours
  text: 'on-device database',             // BM25 keyword relevance
  filter: Filter.eq('lang', 'en') & Filter.gte('year', 2020),
  k: 5,
  fusion: const Fusion.rrf(),             // or Fusion.weighted(0.7)
  mmr: 0.7,                               // diversify near-duplicates
);
for (final h in hits) {
  print('${h.id} ${h.score} ${h.text}');
}
await kb.close();

• Filters use MongoDB semantics — eq, ne, gt/gte/lt/lte, between, inList, notIn, exists, combined with &, | and ~ (or Filter.raw({...}) for a JSON filter document). Arrays match when any element does; dot paths ('author.name') reach nested objects. Filters are answered from an ordered metadata index, not a scan, and vector search within a filter is exact for selective filters.
• Full text is BM25 over a Unicode tokenizer with CJK bigrams, stopwords and conservative plural stemming.
• Atomic writes — an upsert batch lands entirely or not at all; vectors are on disk before the documents commit, and a crash in between is repaired on the next open.
• A collection without dimensions is a text + metadata store.

Reactive queries

Watch committed changes instead of polling. A Stream ends when it is cancelled or the database closes, and each subscription runs its blocking native poll on its own isolate, so a UI isolate never blocks:

final db = await AsyncPhoenixDB.open('app.pdb');

final sub = db.changes(prefix: utf8Key('user:'), values: true).listen((c) {
  print('${c.kind.name} ${c.keyString} ${c.valueString ?? ''}');
});

await db.insert(utf8Key('user:1'), utf8Value('ada'));   // put   user:1 ada
await db.delete(utf8Key('user:1'));                     // delete user:1
await sub.cancel();

Collections report document ids, which is what a list view needs:

final kb = await AsyncPhoenixCollection.open('kb', dimensions: 384);
kb.changes().listen((c) => refreshRow(c.id, removed: c.kind == ChangeKind.delete));

• Only committed writes are published, after the log is durable — a rolled-back or conflicted transaction produces nothing, and a change you see can never be taken back by a crash.
• Order is commit order, and Change.commitTs exposes it.
• A slow consumer loses the oldest changes, not the newest: the queue is bounded by capacity, and ChangeWatcher.dropped (or the gap in commitTs) says how many. Re-read rather than trusting the stream when that matters.
• A restore delivers one ChangeKind.reset instead of a change per key.
• Synchronous code can use db.watch(...) / collection.watch(...) directly; poll() blocks the calling isolate.

Maintenance and recovery

final report = kb.verify();          // documents vs vectors, structural check
if (!report.isConsistent) print('re-ingest ${report.missingVectors} documents');
kb.compact();                        // reclaim tombstoned vectors and free pages
kb.backup('kb-backup');              // consistent, compacted copy

// When a file is too damaged to open at all:
final salvaged = PhoenixDatabase.salvage('broken.pdb', 'recovered.pdb');
print('${salvaged.keysRecovered} keys recovered, '
      '${salvaged.pagesDamaged} pages lost');

salvage scans every page, keeps the pairs on intact leaf pages (newer copies of a key winning), and never touches the original. The phoenixdb_salvage binary does the same from a shell, and phoenixdb_verify reports damage without changing anything.

AI toolkit: RAG, embeddings and LLM clients

package:phoenixdb/ai.dart turns a collection into a retrieval-augmented generation pipeline. Everything is behind small interfaces (ChatModel, Embedder, DocumentStore), so providers are interchangeable:

import 'package:phoenixdb/ai.dart';
import 'package:phoenixdb/phoenixdb.dart';

final kb = await AsyncPhoenixCollection.open('kb', dimensions: 1024);
final rag = RagPipeline(
  store: kb,
  embedder: OpenAICompatibleEmbedder.voyage(
      apiKey: voyageKey, model: 'voyage-3.5', dimensions: 1024),
  chat: AnthropicChatModel(apiKey: anthropicKey),   // claude-opus-5
);

await rag.ingest(RagDocument('handbook', handbookText, metadata: {'team': 'hr'}));

final answer = await rag.ask('How many vacation days do I get?');
print(answer.text);                       // "... 25 days [1]."
for (final source in answer.citations) {
  print('[${source.number}] ${source.documentId}: ${source.text}');
}

// Or stream it:
final streamed = await rag.askStream('And sick leave?');
await streamed.text.forEach(stdout.write);

Component	What it does
AnthropicChatModel	Claude via the Messages API: streaming, adaptive thinking, prompt caching, server-side refusal fallbacks
OpenAICompatibleChatModel	OpenAI, Ollama, Gemini, vLLM, LM Studio, Mistral, Groq, OpenRouter, …
OpenAICompatibleEmbedder	OpenAI, Voyage AI, Ollama and other /embeddings servers, batched
HashingEmbedder	Offline, deterministic lexical embeddings for tests and fully offline apps
CachedEmbedder	Never embeds (or pays for) the same text twice; persists vectors in a PhoenixDatabase
TextChunker	Boundary-aware chunking with overlap and exact source offsets
RagPipeline	Chunk → embed → upsert; hybrid retrieval with MMR; grounded, cited answers
SemanticCache	Reuses answers to prompts that mean the same thing; cache.wrap(model)
ConversationMemory	Stores a conversation and recalls relevant older turns for long chats
Tool + Agent	Tool calling on both providers, and a bounded agent loop that runs the tools the model asks for
knowledgeBaseTool	Hands the model a search tool over the local collection, so it can look things up itself

The clients use dart:io and add no dependencies. Refusals surface as LlmRefusalException, HTTP failures as LlmHttpException (rate limits and overload are retried with backoff first), and a stream that ends before the provider's terminal event is reported rather than passed off as a short answer.

Agents: let a model use your code and your data

final agent = Agent(
  model: AnthropicChatModel(apiKey: key),   // or any OpenAI-compatible server
  tools: [
    knowledgeBaseTool(rag),                 // search the on-device collection
    Tool(
      name: 'create_invoice',
      description: 'Create an invoice for a customer and return its number.',
      schema: {
        'type': 'object',
        'properties': {
          'customer': {'type': 'string'},
          'amount': {'type': 'number'},
        },
        'required': ['customer', 'amount'],
      },
      run: (input) async => '${await createInvoice(input)}',
    ),
  ],
  maxSteps: 6,
);

final run = await agent.run('Invoice Acme for last month per our rate card');
print(run.answer);
for (final step in run.steps) print(step);   // every call, in order

The loop is inspectable and bounded: every step is recorded, a tool that throws is reported to the model (which can try something else) instead of failing the run, an unknown tool name is answered with the real names, and maxSteps stops a confused model from spinning. AgentRun.transcript continues the conversation later.

Observability

PhoenixDB emits lifecycle trace events and an engine metrics snapshot, both reachable from Dart without a separate monitoring dependency:

final db = PhoenixDatabase.open('app.pdb');
db.insert(utf8Key('k'), utf8Value('v'));
db.get(utf8Key('k'));

// A human-readable metrics snapshot covering WAL latency percentiles,
// cache hit ratio, and LSM compaction stats.
print(db.metricsReport());

Trace events are streamed through TraceListener:

final db = PhoenixDatabase.open('app.pdb');
db.setTraceListener((event) => print(event));
// open(path=app.pdb)
// insert(key=k, txn=0)
// get(key=k, txn=0)

Metric	Source
WAL fsync p50/p99/p999	commit latency histogram
Read latency p50/p99	get / scan histograms
Cache hit ratio	page cache hit/miss counters
Compaction throughput	LSM compaction byte counters

Security model

• FFI guardrails — every entry point validates pointer non-nullness and length limits (key ≤ 1 MiB, value ≤ 10 MiB) before any dereference, and returns -2 for violations.
• Handle tagging — each handle carries a magic tag checked in constant time, so use-after-free and foreign pointers are rejected instead of dereferenced.
• Constant-time comparison — security::ct_eq / ct_eq_u64 fold differences with bitwise XOR masking, with no early exit for equal-length inputs.
• Panic containment — every FFI body runs inside catch_unwind; a panic becomes status -7 and never unwinds into Dart.
• Single free path — Rust-allocated memory is released only by phoenix_buffer_free / phoenix_string_free, wired into Dart's NativeFinalizer.

Status codes

Code	Meaning
0	Success
-1	Unclassified error
-2	Invalid argument (null pointer / length limit)
-3	Key not found
-4	Corruption (CRC mismatch)
-5	I/O error
-6	Write-write conflict — retry
-7	Panic caught at the boundary
-8	Unknown transaction
-9	Capacity exceeded
-10	Transaction aborted (e.g. after a failed commit)
-11	Busy — the file is locked by another process

Testing

cd rust && cargo test --features full   # unit, FFI safety, durability, crash and salvage tests
dart test                               # sync, async, reactive, collection and AI toolkit tests
dart analyze --fatal-infos     # clean

Fuzzing

Three libfuzzer harnesses live in rust/fuzz/:

cargo +nightly fuzz run fuzz_btree -- -max_total_time=60   # vs a BTreeMap oracle
cargo +nightly fuzz run fuzz_page  -- -max_total_time=60   # arbitrary page bytes
cargo +nightly fuzz run fuzz_ffi   -- -max_total_time=60   # hostile FFI inputs

Limits

Limit	Value
Page size	4096 bytes
Max key (FFI)	1 MiB
Max key (B+Tree structural)	1 KiB
Max value	10 MiB
Max inline value	1 KiB (larger spills to overflow pages)
Max vector dimensions	65 536
Max vector id	128 bytes
Max k per search	4096 (collections: 10 000)
Max watch queue	1 048 576 changes per subscription
Longest metadata field path	256 bytes (indexed and queryable)
Max document (text + metadata)	8 MiB
Indexed metadata values per document	1024
Longest indexed metadata string	512 bytes (longer strings are stored, not filterable)

License

BSD-3-Clause