WolfDefenderTokenizer class final
The tokenizer of Wolf Defender Edge: the Gemma-style BPE of its base
model, mmBERT. It gives the same IDs as Hugging Face tokenizers for its
tokenizer.json, without special tokens.
- Added tokens split the text first: the leftmost match, and at one
position the longest. They match the original text. Examples are
<bos>,\n\n,\t\tand<b>. - Each other part: " " becomes "▁", and "▁" is prepended if the part does not start with it. The part splits before each "▁".
- BPE on each word, by merge rank. A character that is not in the
vocabulary becomes its UTF-8 byte tokens (
<0x0A>), or<unk>if a byte token is missing. Adjacent unknowns become one<unk>.
The file, little-endian: magic, version, the token count and each token
(UTF-8 byte length as uint16, then the bytes; IDs 0, 1, 2, ...), the merge
count and each merge in rank order (left ID, right ID; uint32), the added
token count and each added token ID (uint32).
tool/wolf_defender/convert_tokenizer.dart writes it.
Constructors
- WolfDefenderTokenizer.fromBytes(Uint8List bytes)
-
factory
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- unkId → int
-
final
- vocabularySize → int
-
no setter
Methods
-
encode(
String text) → List< int> -
The IDs of
text, without special tokens at the ends. -
id(
String token) → int? -
The ID of
token, or null. -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
token(
int id) → String -
The token with
id. -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited