fire_rag 1.3.0

Firebase RAG for AI Agents

fire_rag

fire_rag is a Firebase-first RAG ingestion and retrieval helper built around Firestore vector records.

It combines:

• resumable Cloud Tasks orchestration from arcane_admin
• chunking, embeddings, and distillation models from agentic
• Firestore vector storage that can later be queried directly or through rag

The package is centered around one idea: treat a repository record as something that can be re-vectorized safely, tracked while it is ingesting, and queried later with generation-aware filtering so stale vectors do not leak back into retrieval.

What 1.3.0 Adds

1.3.0 expands fire_rag from a task-only ingestion package into a repository-oriented workflow.

The new surface includes:

• Repository and RepositoryRecord mixins for your app models
• ChunkSource implementations for bucket files, Firestore documents, and raw strings
• FireRag.pushRecordVectors(...) for scheduling a full record refresh
• FireRag.query(...) and FireRag.querySingle(...) for vector search scoped to a repository
• generation-aware cleanup and filtering so old vectors can be invalidated cleanly
• progress tracking fields for core and distilled vector work
• image-aware embedding support when chunk content points at storage-backed image paths

Concept

fire_rag assumes your application has a repository of records that should become searchable.

A typical flow looks like this:

1. Model a repository and record in your own app.
2. Have the record expose one or more ChunkSource values.
3. Call FireRag.instance.pushRecordVectors(repository, record).
4. fire_rag schedules chunk, embed, and distill tasks.
5. Chunk docs are written into your vector collection with repository and record metadata.
6. Later, call FireRag.instance.query(...) to retrieve the best matching chunks for that repository.

This makes fire_rag useful as both:

• an ingestion pipeline
• a lightweight retrieval facade over the stored vectors

Installation

dart pub add fire_rag

Typical companion packages are:

dart pub add arcane_admin
dart pub add agentic
dart pub add rag

If you change artifact-backed models, regenerate code before publishing:

dart run build_runner build --delete-conflicting-outputs

Initialization

FireRag.init(...) now needs both the Firestore vector collection name and the storage bucket used for storage-backed chunk content.

import 'package:agentic/agentic.dart';
import 'package:arcane_admin/arcane_admin.dart';
import 'package:fire_rag/fire_rag.dart';

Future<void> main() async {
  await ArcaneAdmin.initialize(
    projectId: 'my-project-id',
    defaultStorageBucket: 'my-project-id.firebasestorage.app',
  );

  ConnectedEmbeddingModel embedder = OpenAIConnector(
    apiKey: const String.fromEnvironment('OPENAI_API_KEY'),
  ).asEmbedder('text-embedding-3-small');

  ConnectedChatModel llm = OpenAIConnector(
    apiKey: const String.fromEnvironment('OPENAI_API_KEY'),
  ).connect(ChatModel.openai4_1Mini);

  FireRag.init(
    embed: embedder,
    llm: llm,
    taskQueue: 'rag-ingest',
    endpointUrl: 'https://your-service.run.app/event/executeJob',
    vectorCollection: 'rag_vectors',
    bucket: 'my-project-id.firebasestorage.app',
  );
}

Model Your Data

The new usage pattern starts by making your application models implement Repository and RepositoryRecord.

import 'package:artifact/artifact.dart';
import 'package:fire_rag/fire_rag.dart';
import 'package:fire_rag/record/repository.dart';
import 'package:fire_rag/task/task_chunk.dart';

@artifact
class KnowledgeBase with Repository {
  @override
  final String name;

  @override
  final String repositoryPath;

  const KnowledgeBase({
    required this.name,
    required this.repositoryPath,
  });
}

@artifact
class KnowledgeRecord with RepositoryRecord {
  @override
  final String recordPath;

  @override
  final String generation;

  @override
  final String? ingestionError;

  @override
  final ProgressTrackerState coreVectorProgress;

  @override
  final ProgressTrackerState distilledVectorProgress;

  final String bucket;
  final String path;

  const KnowledgeRecord({
    required this.recordPath,
    required this.generation,
    required this.bucket,
    required this.path,
    this.ingestionError,
    this.coreVectorProgress = const ProgressTrackerState(total: 0),
    this.distilledVectorProgress = const ProgressTrackerState(total: 0),
  });

  @override
  Stream<ChunkSource> getSources() => Stream.value(
    ChunkBucketSource(bucket: bucket, path: path),
  );
}

fire_rag uses these fields for:

• repository scoping during retrieval
• generation validation during query-time filtering
• progress updates during core and distilled embedding work
• source enumeration when refreshing a record

Choose A Source Type

TaskChunk no longer assumes every source is a bucket path.

You can feed ingestion from:

• ChunkBucketSource Downloads a file from Cloud Storage.
• ChunkDocumentSource Reads a Firestore document and serializes it to JSON before chunking.
• ChunkStringSource Chunks a raw string directly.

Examples:

ChunkBucketSource(bucket: 'docs', path: 'articles/a.txt')

ChunkDocumentSource(
  documentPath: 'knowledge/articles/a1',
  fieldMask: ['title', 'body', 'summary'],
)

ChunkStringSource(source: 'Inline text that should be embedded')

Ingest Or Refresh A Record

For most applications, the new top-level entry point is pushRecordVectors(...).

await FireRag.instance.pushRecordVectors(repository, record);

What this does:

1. Generates a fresh ULID generation ID.
2. Schedules one TaskChunk per source returned by record.getSources().
3. Tags every written vector record with repository, record, and generation metadata.
4. Updates the source record's generation field.
5. Deletes stale vectors for the same record from older generations.

This gives you a clean "replace the searchable representation of this record" workflow without having to manually manage old vector rows.

Query A Repository

You can now query the stored vectors directly through FireRag.

Single Query

FireResultSet results = await FireRag.instance.querySingle(
  repository: repository,
  query: Content.text('What changed in the onboarding flow?'),
  maxResults: 10,
);

Multi Query

FireResultSet results = await FireRag.instance.query(
  repository: repository,
  queries: [
    Content.text('onboarding changes'),
    Content.text('signup funnel updates'),
  ],
  maxResults: 20,
  maxHits: 100,
);

Each result is a FireResult:

• queryIndex: which input query produced the hit
• chunk: the matched Chunk
• score: vector similarity score

query(...) merges and sorts results across multiple query variants.

If validateGeneration is left enabled, results are filtered so vectors from older generations of the same record are skipped automatically.

Advanced Direct Task Scheduling

If you want lower-level control, you can still schedule TaskChunk directly.

await FireRag.instance.taskManager.schedule(
  TaskChunk(
    taskId: 'chunk.article.123',
    source: ChunkBucketSource(
      bucket: 'docs',
      path: 'articles/article-123.txt',
    ),
    recordLocation: const RecordLocation(
      repositoryPath: 'repositories/knowledge',
      recordPath: 'records/article-123',
    ),
    destinationCollection: 'rag_vectors',
    bucket: 'my-project-id.firebasestorage.app',
    distillationFactor: 4,
    destinationMetadata: {
      'kind': 'article',
      'generation': '01JABCXYZ...',
    },
  ),
);

This path is useful when you want to integrate with your own eventing layer but still reuse the same task pipeline.

Progress Tracking

fire_rag now updates record-level progress fields while ingestion runs:

• coreVectorProgress.total
• coreVectorProgress.completed
• distilledVectorProgress.total
• distilledVectorProgress.completed

Core progress is updated as base chunks are embedded. Distilled progress is updated as higher-LOD chunks are scheduled and embedded.

That makes it straightforward to show ingestion status in your own app UI.

Storage Shape

Vector documents are stored in the serialized agentic.Chunk shape, with app routing metadata living inside metadata.

Typical metadata includes:

• record
• repository
• generation

Document IDs are derived from a stable hash of repositoryPath and recordPath, then suffixed with level and chunk index:

{recordHash}.L{lod}.{index}

This keeps vector IDs stable across retries while still separating multiple levels of detail.

Image Support

TaskEmbed now supports image-aware embedding for chunks whose modality is image.

In that case:

• chunk.content is treated as a storage path
• the file is read from the configured bucket
• the image is converted to base64
• the embedding model receives image content instead of plain text

Text chunks still embed chunk.fullContent as before.

Relationship To Other Packages

fire_rag stays intentionally small by leaning on a few focused packages:

• agentic Provides Chunk, Content, chunking, embeddings, and connected chat models.
• arcane_admin Provides Firebase admin access plus the resumable TaskManager execution model.
• rag Acts as the broader retrieval-side companion package if you want to build a fuller retrieval stack around the same stored vectors.
• fire_api Backs Firestore, Storage, vector fields, and nearest-neighbor search primitives.
• artifact Handles serialization for tasks and artifact-backed models.

Notes

• pushRecordVectors(...) is now the preferred application entry point.
• TaskChunk remains useful when you need explicit low-level orchestration.
• Distillation is optional. Leave distillationFactor unset if you only want core chunks plus embeddings.
• Query-time generation validation is enabled by default to avoid returning stale vectors after a record refresh.
• The README examples assume your own record documents already expose the progress and generation fields that fire_rag updates.

Contributing

If you change task models or other artifact-backed state, regenerate code before publishing:

dart run build_runner build --delete-conflicting-outputs