BpeTokenizer class

Constructors

BpeTokenizer.fromJson(String json)

Parse a tokenizer from a JSON string produced by saveJson.

factory

BpeTokenizer.loadFile(String path)

Load a tokenizer from a file.

factory

BpeTokenizer.train(String corpus, {required int targetVocabSize, int minCount = 2})

Train a fresh tokenizer from a text corpus.

factory

Properties

hashCode → int

The hash code for this object.

no setter, inherited

merges → List<List<int>>

Ordered list of merges. merges[i] = (a, b) means "when encountering the pair (a, b), replace it with token id 256 + i".

final

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

vocabSize → int

Total vocabulary size (256 base bytes + number of merges).

final

Methods

decode(List<int> tokens) → String

Decode a list of token ids back to a string. Unpacks each merged id recursively down to raw byte ids, then UTF-8-decodes.

encode(String text) → List<int>

Encode a string into a list of token ids. Applies merges in the order learned during training (lowest rank first).

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

saveFile(String path) → void

Write the tokenizer to a file.

saveJson() → String

Serialize the tokenizer to a compact JSON string.

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited