CharTokenizer class

Constructors

CharTokenizer.fromJson(String s)

factory

CharTokenizer.fromText(String text)

Fit a tokenizer to text.

factory

CharTokenizer.fromVocab(List<String> chars)

Restore from a previously-saved char list. chars must contain unique entries; order defines the id mapping.

factory

CharTokenizer.loadFile(String path)

factory

Properties

chars → List<String>

Vocabulary in canonical order — chars[i] is the string for id i.

final

hashCode → int

The hash code for this object.

no setter, inherited

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

vocabSize → int

no setter

Methods

decode(List<int> ids) → String

Decode a list of int ids back to a string. Out-of-range ids become the empty string.

encode(String s) → List<int>

Encode a string to a list of int ids. Unknown chars → 0.

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

saveFile(String path) → void

toJson() → String

JSON wire format:

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited