core/data/hf_bpe_tokenizer library
Loads a HuggingFace tokenizer.json (byte-level BPE, e.g.
GPT-2, GPT-NeoX/Pythia, Llama-style) and provides encode/decode.
The algorithm mirrors what tokenizers does at runtime:
- UTF-8 byte encode the input string.
- Byte→unicode map (GPT-2's
bytes_to_unicode): converts each byte 0-255 to a printable single-code-point string, dodging whitespace and control chars. Bytes we recognise as "safe printable" (0x21-0x7e, 0xa1-0xac, 0xae-0xff) pass through as-is; the rest are shifted into the private range starting at 0x100. Ġ = U+0120 corresponds to byte 0x20 (SPACE). - Pre-tokenizer regex (GPT-2 default) to chop the byte-mapped string into "words".
- BPE merges: for each word (as a sequence of single-char symbols), iteratively fuse the adjacent pair with the lowest merge rank until no more merges apply.
- Vocab lookup to turn each final symbol into its int id.
Decode reverses steps 5→2 and UTF-8 decodes the resulting bytes.
Supports:
- loading tokenizer.json for GPT-2, distilgpt2, Pythia (any
size), GPT-Neo — anything with
model.type == "BPE"and aByteLevelpre-tokenizer. - added special tokens (e.g.
<|endoftext|>) — matched as literal substrings before pre-tokenizing.