knowledge_embeddings library
A library for chunking, embedding, and searching content.
This library provides a unified API for working with different types of content (code, documents, etc.) and different embedding strategies.
Classes
- BaseChunker
- Base interface for all content chunkers.
- BaseEmbedder
- Base interface for all content embedders.
- BaseStore
- Base interface for all content stores.
- BM25LexicalIndex
- Exact in-memory BM25 index over chunk text.
- Chunk
- Base class for all content chunks.
- ChunkerRegistry
- Registry for content chunkers.
- ChunkMetadata
- Metadata for chunking content.
- ContentSearcher
- A class for searching content using embeddings.
- DartChunker
- Embedding
- Represents an embedding for a chunk of content.
- EmbeddingModelSpec
- Pinned artifact and preprocessing contract for a local embedding model.
- HybridContentSearcher
- Combines exact BM25 lexical search with dense/vector search using RRF.
- IngestionPipeline
- A lightweight orchestrator that chunks files, prepares embeddings, and persists both chunks and vectors.
- LlamaEmbedder
- In-process embeddings from a verified local GGUF. Never downloads at runtime.
- MarkdownChunker
- A chunker for Markdown documents.
- MemoryStore
- An in-memory implementation of BaseStore.
- ObjectBoxStore
- ObjectBox-backed BaseStore implementation with persistent chunk and embedding storage using HNSW vector search indices.
- ParentChildResolver
- Resolves precise child search hits to larger parent/context chunks.
- ParentChildSearcher
- Wraps a precise child Searcher and returns expanded parent context.
- ParentChildSearchResult
- A search hit plus the larger context chunk selected for presentation.
- ReciprocalRankFusion
- Rank-based fusion for combining multiple retrieval result lists.
- RerankingContentSearcher
- Wraps a first-pass Searcher with a cross-encoder reranking stage.
- Searcher
- One ranked-retrieval entry point.
- SearchOptions
- Options for content search.
- SearchReranker
- Reorders first-pass search results with a query/document reranking model.
- SearchResult
- Represents a search result.
- TeiReranker
-
Reranker client for Hugging Face Text Embeddings Inference
/rerank. - TextChunker
- A chunker for plain text documents.
- TypeScriptChunker
- A pure-Dart structural chunker for TypeScript code.
Enums
- LongInputPolicy
- How to handle text longer than the model's tokenizer context.
Constants
- localEmbeddingModel → const EmbeddingModelSpec
- Arctic Embed XS Q8_0, using CLS pooling and normalized retrieval vectors.
Functions
-
chunkFiles(
ChunkerRegistry registry, Iterable< File> files, {Map<String, String?> ? contentTypes, void onFileProcessed(File file, List<Chunk> chunks)?, FileSkippedCallback? onFileSkipped}) → Iterable<ChunkedFile> -
Chunks
fileswithregistry, without an embedder or a store. -
defaultEmbeddingModelFile(
) → File -
Uses the packaged model next to an executable, or
models/during development. -
deterministicChunkId(
{required String sourcePath, required int lineStart, required int lineEnd, required String content, required String type}) → String - Creates a deterministic chunk identifier based on stable attributes.
-
inferContentType(
String path) → String -
Infers a content type identifier from the file extension of
path. -
objectBoxCosineDistanceToSimilarity(
double distance) → double - Converts ObjectBox cosine distance into cosine similarity.
-
tokenizeLexicalText(
String text) → Iterable< String> - Tokenizes text for exact BM25 lexical scoring.
-
validateObjectBoxVectorDimension(
int dimension, {String name = 'dimension', String? context}) → void - Validates that a vector can be stored or queried by the current ObjectBox HNSW schema.
Typedefs
-
ChunkedFile
= ({List<
Chunk> chunks, File file}) - One file and the chunks a chunker produced for it.
- ChunkerMissingHandler = void Function(String inferredType)
- Callback invoked when no chunker is registered for the inferred content type.
- DuplicateChunkCallback = void Function(Chunk chunk)
- Callback invoked when a duplicate chunk is detected.
- FileSkippedCallback = void Function(File file, {String? inferredType, String? reason})
- Callback invoked when a file is skipped during chunking.
-
TeiRerankTransport
= Future<
Object?> Function(Uri uri, {required String query, required Duration requestTimeout, required List<String> texts}) - Sends one TEI rerank request and returns the decoded JSON response.
Exceptions / Errors
- RerankerException
- Exception thrown when a reranker runtime call fails.