transform method

Stream<EmbeddedChunk> transform(Stream<String> rawFeed, { int semaphoreBuffer = 4, })

Transforms a stream of raw text into a stream of embedded chunks.

This method chunks the input text, processes each chunk in parallel (controlled by the semaphore buffer), and generates embeddings for each chunk.

@param rawFeed The input stream of text @param semaphoreBuffer The maximum number of concurrent embedding operations (default: 4) @return A stream of chunks with their corresponding vector embeddings

Implementation

Stream<EmbeddedChunk> transform(
  Stream<String> rawFeed, {
  int semaphoreBuffer = 4,
}) => chunker
    .transformWithOverlap(rawFeed, overlap: overlap)
    .semaphoreMap<EmbeddedChunk>(
      semaphoreBuffer,
      (i) async =>
          EmbeddedChunk(chunk: i, embedding: await embedder(i.content)),
    );