core/data/bpe_tokenizer library

Byte-level Byte Pair Encoding tokenizer.

A pure-Dart implementation that operates on the UTF-8 bytes of the input text — every text is representable, and the initial vocab is exactly the 256 possible byte values. Training then greedily merges the most frequent adjacent pair of symbols until the vocab reaches targetVocabSize (or no merges are possible).

This is not the fastest BPE trainer in existence — the training loop is O(N * merges) in the worst case — but the intent is a small, dependency-free reference implementation suitable for toy-scale training (~1e6 bytes, vocab ~256..2048).

Wire format for save / load:

{
  "version": 1,
  "kind": "byte-bpe",
  "vocabSize": <int>,
  "merges": [[<int>, <int>], ...]   // in merge order, pair -> vocabSize-1
}

Classes

BpeTokenizer