overlap method

Stream<Chunk> overlap(int amount)

Creates a new stream where each chunk overlaps with the previous chunk's content.

This is useful for maintaining context between chunks when processing text for LLM or embedding systems. The overlap helps preserve semantic meaning across chunk boundaries.

@param amount The maximum number of characters to overlap between chunks @return A stream of chunks with overlapping content

Implementation

Stream<Chunk> overlap(int amount) async* {
  Chunk? previous;

  await for (Chunk i in this) {
    if (previous != null) {
      int l = 0;
      List<String> j = [];
      for (String i in previous.content.split(" ").reversed) {
        if (l + i.length > amount) {
          break;
        }

        l += i.length + 1;
        j.add(i);
      }

      String tail = j.reversed.join(" ");
      yield Chunk(
        i.id,
        i.start - tail.length,
        i.length + tail.length,
        "$tail${i.content}",
      );
    } else {
      yield i;
    }

    previous = i;
  }
}