core/data/text_token_dataset library

Language-model sliding-window dataset.

Reads an entire text file (or a plain string), tokenizes it with a caller-supplied tokenizer, and exposes one training example per valid starting index: for each position i, the input is tokens[i .. i + blockSize) and the target is tokens[i + 1 .. i + blockSize + 1) (shifted-by-one next-token prediction, as in GPT-style causal LM training).

The dataset length is therefore numTokens - blockSize. Each item is a LmSample of two [blockSize] Tensors on the requested device.

The tokenizer only needs to expose List<int> encode(String) — both CharTokenizer and BpeTokenizer satisfy this.

Classes

LmSample

One (input, target) pair for next-token prediction training.

TextEncoder

The interface TextTokenDataset needs from a tokenizer. Both CharTokenizer and BpeTokenizer satisfy this via structural typing (Dart uses subtyping, so we accept anything with an encode method matching this shape).

TextTokenDataset