models/wolfDefender/src/wolf_defender_tokenizer library

Classes

WolfDefenderTokenizer

The tokenizer of Wolf Defender Edge: the Gemma-style BPE of its base model, mmBERT. It gives the same IDs as Hugging Face tokenizers for its tokenizer.json, without special tokens.

Constants

metaspace → const String

The SentencePiece word boundary, U+2581.

wolfDefenderTokenizerMagic → const int

"WDTK", little-endian.

wolfDefenderTokenizerVersion → const int