core/data/wordpiece_tokenizer library

BERT WordPiece tokenizer (uncased BERT / MiniLM / all-MiniLM-L6-v2 style). Faithful enough to reproduce HuggingFace BertTokenizer output on the vast majority of English text.

Pipeline (matches BertTokenizer with do_lower_case=true):

  1. Clean text: strip control chars, collapse whitespace.
  2. Lowercase; strip common accents (NFD -> drop Mn).
  3. Split punctuation into their own tokens.
  4. Whitespace-split into words.
  5. For each word: greedy-longest WordPiece match against the vocab, with ##-prefixed continuation pieces; unknown -> [UNK].

Not implemented (kept simple; falls back to sensible defaults):

  • Chinese-character forced splitting.
  • never_split list.
  • Special-token protection (they are recovered by the vocab lookup itself).

Functions

debugPreview(List<int> ids, int max) → String
Utility for callers: List<int> -> jsonEncode(...) doesn't hit arg limits, but a helper avoids duplicating this everywhere in bin/ demos.