TextChunker class

Splits text into chunks of at most chunkSize characters, preferring natural boundaries: paragraphs, then lines, then sentences, then words.

Each chunk ends at the coarsest boundary found in the back half of its window, and the next chunk starts about overlap characters earlier (snapped to the start of a word), so a sentence cut by a boundary still appears intact in one of the two. Offsets always refer to the original text, so a hit can be highlighted in place.

Constructors

TextChunker({int chunkSize = 1000, int overlap = 150, List<String> separators = const ['\n\n', '\n', '. ', '? ', '! ', '。', '？', '！', '; ', ', ', ' ']})

Creates a chunker.

Properties

chunkSize → int

Longest chunk, in characters.

final

hashCode → int

The hash code for this object.

no setter, inherited

overlap → int

Characters of context repeated between consecutive chunks.

final

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

separators → List<String>

Boundaries to cut at, coarsest first. When none fits, a chunk is cut at chunkSize (never inside a surrogate pair).

final

Methods

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

split(String text) → List<TextChunk>

Splits text; whitespace-only input gives no chunks.

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited