wayfinder_embeddings 0.2.0

Chunking, embedding, and retrieval for Wayfinder and other Dart applications.

Unreleased

• BREAKING: Dense-only retrieval replaces the former BM25/hybrid paths. Supply embedder: to both KnowledgeIndex constructors; remove KnowledgeRetrievalMode, search mode:, and OKF candidateLimit:. Remove lexical tokenization, BM25, hybrid searcher, and reciprocal-rank-fusion APIs without deprecated aliases. Use ContentSearcher instead.
• Keep generic Searcher, reranking, rerank:dense, TEI, and reranker candidate limits. Evaluation defaults to dense; retired descriptors fail before work. Actual evaluators always load the model; remove their --native toggle and the knowledge example's --dense toggle. Basic/code examples and validation are explicitly preview-only. Fixture gates remain structural; native CI owns genuine dense relevance checks against an extracted historical dense baseline.
• Fail actual OKF search when eligible passages lack compatible vectors. Generic search detects an empty compatible embedding space without scanning successful ANN queries; callers still own partial vector coverage. No model, snapshot, input identity, storage schema, context, or chunking change is implied by this cutoff. Compatible saved indexes remain usable.

0.2.0

• OkfLinkField names a producer frontmatter key whose entries are typed links, and okfFieldEdges resolves them as okf resolves a link target. Both live in package:wayfinder, which this package now depends on; okf_knowledge.dart no longer exports them. KnowledgeSnapshot.load, fromSources, and fromMap accept linkFields; search expansion follows the resulting KnowledgeSnapshot.fieldEdges before okf's untyped edges and reports each on KnowledgeContextHit.fieldEdge. Without link fields nothing changes.

Breaking API placement cleanup with a hard cutoff; no deprecated aliases or compatibility wrappers are provided:

• Replace EmbeddingEntity.computeKey(...) with embeddingIdentityKey(...).
• Replace chunkFiles(registry, files, ...) with registry.chunkFiles(files, ...).
• Replace deterministicChunkId(...) with Chunk.computeId(...).
• Replace ReciprocalRankFusion.defaultRankConstant with defaultReciprocalRankConstant, and ReciprocalRankFusion.fuse(...) with fuseReciprocalRanks(...).
• Replace RetrievalEvaluator.evaluate(...) with RetrievalEvaluation.evaluate(...), and .evaluateQuery(...) with QueryRetrievalEvaluation.evaluate(...).
• Replace RetrievalRegressionGate.evaluate(...) with RetrievalRegressionGateResult.evaluate(...).
• Replace prepareFixtureConfig(...) with FixtureConfig.prepare(...), and runPreview(...) with PreviewResult.fromFiles(...).
• Replace RetrievalEvaluator.summarizeGroups(...) with the top-level summarizeRetrievalQueryGroups(...) orchestration helper.
• Replace ObjectBoxStore(...) with ObjectBoxStore.open(...).
• Replace KnowledgeIndex.openSnapshot(...) with KnowledgeIndex.fromSnapshot(...).

0.1.1

• Recover oversized embedding inputs with lossless Unicode-safe subdivision and conditional derived-context omission in the shared OKF fitter.
• Keep whitespace attached to nonblank text and recheck the full token budget, avoiding blank-input failures after context omission or in body-only fitting.
• Persist effective inputs and original-chunk recovery attribution; expose immutable aggregated diagnostics across reopening and tighter-budget refitting. Source files, strict encoder rejection, model identity, and storage schema are unchanged.

0.1.0

• Depend on okf ^0.5.0, which reads every timestamp as an ISO 8601 datetime with an explicit UTC offset and compares stale_after as an instant rather than a local calendar day.
• Raise the Dart floor to 3.11.0, matching every Concepta package. The pinned llamadart 0.8.23 requires 3.10.7, so the declared floor now sits above what the dependencies strictly need.

0.0.1

Stable release for the Wayfinder package split; retrieval APIs are unchanged.

Changelog

0.0.1-dev.0 — wayfinder_embeddings

• Introduce the Wayfinder library package, replacing knowledge_embeddings.
• Preserve existing ObjectBox databases using compatible old/new schema markers.
• Add WAYFINDER_EMBEDDING_MODEL with the legacy override as a fallback.
• Reuse verified legacy model caches, including offline preparation.

History before the package rename

The entries below describe releases of knowledge_embeddings.

0.0.1-dev.1

• Update library documentation and application links for the Wayfinder rename.

0.0.1-dev.0

• Verifies the pinned ObjectBox release archive by SHA-256 before any of its bytes reach the package, replacing the upstream download script, with a pinned hash for every platform upstream publishes and no architecture fallback.

• Keys stored vectors on the model's bytes and preprocessing contract only, so another mirror of the same verified artifact reuses them.

• Retries a cold native backend start once, because the first load from a freshly installed bundle can exceed the runtime's worker startup timeout, and reports the retry count through LlamaEmbedder.coldStartRetries.

• Chunks Markdown footnote definitions as footnote apparatus and keeps them out of OKF passages, where attribution resolves through sources.

• Regenerates raw per-process benchmark reports instead of checking in multi-megabyte archives; summaries and provenance stay in fixtures/.

• Records the accepted local model/defaults and measured improvement priorities in ADR-0009, with available CLI commands and their current scope.

• Adds repeatable Arctic XS/S and BGE-small model comparisons, with pinned Q8/Q4 experimental artifacts, per-model tokenizer validation, and isolated indexes.

• Adds isolated keyword/BM25/dense/hybrid benchmarks over 160 topic-separated questions, plus compiled startup/cache/update, build, and 1k–50k storage sweeps.

• Shares the lexical tokenizer and canonical context assembly with diagnostic rankers, preserving scope and original citation text for external rankings.

• Adds an upstream-OKF adapter with original citations, contextual token-budget splitting, explicit lifecycle/scope/governing/link policy, and atomic bundle synchronization that reuses unchanged vectors.

• Measures seven retrieval configurations over separate development and held-out fixture queries with BM25, local dense, and hybrid modes on both stores.

• Chunks Dart, TypeScript, Markdown, and text files into stable, addressable segments with line ranges and symbol metadata.

• Ranks chunks with exact BM25, local llamadart vectors, and Reciprocal Rank Fusion, with optional reranking and parent/child expansion.

• Stores chunks and embeddings in MemoryStore or the optional 384-dimension ObjectBoxStore.

• Ships a 65-query qrels fixture, a checked-in BM25 metrics baseline, and a regression gate test over fixtures/corpus.

• Keeps reusable example inputs in fixtures/samples, with full chunk goldens for both corpora and portable fixture paths and line endings.

• Deduplicates pending ingestion batches, releases benchmark resources after factory failures, and preserves Markdown code/table boundaries and headings.

• Computes stable cosine similarity for very large and very small finite vectors.

• Tokenizes each BM25 document once and removes redundant forwarding and copying in model, embedder, storage, and workflow paths.

• Pins a 25.28 MB Arctic Embed XS Q8_0 model; verifies size and SHA-256, separates query/document encoding, and rejects token overflow by default.

• Stages cached model downloads during preparation and bundles weights, native libraries, manifest, and model license with the retrieval CLI.

• Protects existing databases from incompatible vector-schema changes; reindex source files into a fresh directory.

• Exercises the selected store for lexical benchmarks as well as semantic runs.

• Preserves supplied Markdown metadata and refreshes metadata during ingestion without re-encoding unchanged content.

• Preserves ObjectBox entity IDs on updates, deletes chunks/vectors atomically, rejects invalid float32 vectors, and falls back to exact scoring within a model when ANN filtering leaves too few usable results.

• Adds 13 diagnostic knowledge cases for authority, lifecycle, freshness, paraphrases, negation, explicit scopes, and unanswerable questions.

• Initial package release; provides the retrieval implementation used by Wayfinder.