backfill method

Future<DVSemanticBackfill> backfill(Iterable<T> records, { bool complete = false, })

Embeds records into this index's generation, skipping records a previous run already did, so a failure part-way resumes rather than re-embedding — and re-paying for — the records before it.

With complete, once every record is done the generation becomes the one queries answer from and the previous generation is dropped.

Implementation

Future<DVSemanticBackfill> backfill(
  Iterable<T> records, {
  bool complete = false,
}) async {
  await _open();
  final String gen = generation;
  int processed = 0;
  int total = 0;
  for (final T record in records) {
    total++;
    final String doneKey = 'semantic:done:$gen:${idOf(record)}';
    if (await vectors.readMeta(doneKey) == null) {
      await _write(gen, record, embedder);
      await vectors.writeMeta(doneKey, '1');
    }
    processed++;
  }
  bool completed = false;
  if (complete && processed == total) {
    final String? previous = _active;
    await vectors.writeMeta(_activeKey, gen);
    _active = gen;
    if (previous != null && previous != gen) {
      await vectors.dropIndex(previous);
    }
    completed = true;
  }
  return DVSemanticBackfill(
    generation: gen,
    processed: processed,
    total: total,
    completed: completed || !_rebuilding,
  );
}