RagIndexer class final
Chunks documents, embeds them and writes them to an index.
Three things that make re-ingestion safe
Stable identifiers. A chunk is <documentId>#<index>, so re-indexing a
document replaces its chunks rather than adding a second copy. Retrieval
returning the same passage twice is not a cosmetic problem — it consumes the
prompt budget and makes the model more confident about whatever it says.
Content hashing. An unchanged document is skipped without embedding it again. Re-embedding an unchanged corpus is the largest avoidable cost in a RAG system, and the fingerprint lives in the index itself, so there is no second table to keep in step.
Tail deletion. A document that used to produce nine chunks and now produces six leaves three stale ones behind. They still match queries and still get cited, quoting text that no longer exists. After writing, the indexer deletes everything at or beyond the new chunk count.
final indexer = RagIndexer(
index: embeddingIndex,
chunker: const MarkdownChunker(),
keywordIndex: keywordIndex,
);
final report = await indexer.indexAll(documents);
print('${report.indexed.length} indexed, ${report.skipped.length} unchanged');
Constructors
- RagIndexer({required EmbeddingIndex index, Chunker chunker = const RecursiveChunker(), InMemoryKeywordIndex? keywordIndex, String? namespace, bool skipUnchanged = true, bool storeText = true})
- Creates an indexer.
Properties
- chunker → Chunker
-
How documents are split.
final
- hashCode → int
-
The hash code for this object.
no setterinherited
- index → EmbeddingIndex
-
The embedding model and vector store to write to.
final
- keywordIndex → InMemoryKeywordIndex?
-
A lexical index kept in step, when hybrid retrieval is in use.
final
- namespace → String?
-
The store partition to write to.
final
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- skipUnchanged → bool
-
Whether documents whose content hash is unchanged are skipped.
final
- storeText → bool
-
Whether chunk text is stored alongside its vector.
final
Methods
-
indexAll(
Iterable< RagDocument> documents, {AgenticContext? context}) → Future<IndexingReport> -
Chunks, embeds and writes
documents. -
indexOne(
RagDocument document, {AgenticContext? context}) → Future< ({int removed, bool skipped, int written})> - Indexes one document.
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
remove(
String documentId, {AgenticContext? context}) → Future< int> -
Removes every trace of
documentId, returning how many chunks went. -
toString(
) → String -
A string representation of this object.
override
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited