addSourceUtf8WithChunking method

Future<SourceAddResult> addSourceUtf8WithChunking(List<int> bytes, { String? metadata, String? name, ChunkingStrategy? strategy, Duration? chunkDelay, void onProgress(int done, int total)?, })

Add a document from UTF-8 bytes while avoiding caller-side String inflation.

The bytes are handed directly to Rust, which decodes UTF-8 once and runs the ingest pipeline without round-tripping the body through a Dart String. For a 4 MB UTF-8 buffer this saves the ~8 MB Dart heap UTF-16 allocation and one Rust→Dart→Rust traffic leg compared to the legacy extractTextFromUtf8 → addSourceWithChunking two-step.

Implementation

Future<SourceAddResult> addSourceUtf8WithChunking(
  List<int> bytes, {
  String? metadata,
  String? name,
  ChunkingStrategy? strategy,
  Duration? chunkDelay,
  void Function(int done, int total)? onProgress,
}) async {
  final effectiveStrategy = strategy ?? ChunkingStrategy.recursive;
  final prepared = await rust_ingest.prepareSourceIngestionFromUtf8(
    collectionId: collectionId,
    contentBytes: bytes,
    metadata: metadata,
    name: name,
    strategy: _toIngestStrategy(effectiveStrategy),
    maxChars: maxChunkChars,
    overlapChars: overlapChars,
  );
  return _runPreparedIngestion(
    prepared,
    chunkDelay: chunkDelay,
    onProgress: onProgress,
  );
}