Tokenizer class final
Encode/decode between strings and token ids using a LlamaVocab.
Stateless and cheap to construct. Hold one alongside the model.
Constructors
- Tokenizer(LlamaVocab vocab)
-
const
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- vocab → LlamaVocab
-
final
Methods
-
decode(
int token, {bool unparseSpecial = true}) → String -
Detokenize a single
tokeninto its UTF-8 piece. -
decodeAll(
List< int> tokens, {bool removeSpecial = false, bool unparseSpecial = true}) → String - Detokenize a list of tokens into a UTF-8 string.
-
encode(
String text, {bool addSpecial = true, bool parseSpecial = true}) → List< int> -
Tokenize
text. -
encodeToken(
int token, {bool special = true}) → Uint8List - Raw byte form of a single token's piece — useful when streaming, where piece boundaries do not align with UTF-8 codepoints.
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited