BaseTokenizer class

BaseTokenizer class for tokenization and detokenization.

Implementers

Constructors

BaseTokenizer(List<String> tokens, {int? startingIndex, String initToken = "[CLS]", String eosToken = "[SEP]", String padToken = "[PAD]", String unkToken = "[UNK]"}): Constructor for BaseTokenizer.

bosToken ↔ String: getter/setter pair
bosTokenId ↔ int: getter/setter pair
eosToken ↔ String: getter/setter pair
eosTokenId ↔ int: getter/setter pair
hashCode → int: The hash code for this object.
no setterinherited
i2s ↔ Map<int, String>: getter/setter pair
length → int: no setter
padToken ↔ String: getter/setter pair
padTokenId ↔ int: getter/setter pair
runtimeType → Type: A representation of the runtime type of the object.
no setterinherited
s2i ↔ Map<String, int>: getter/setter pair
unkToken ↔ String: getter/setter pair
unkTokenId ↔ int: getter/setter pair

detokenize(List<int> tokens) → String: Detokenizes the tokens.
noSuchMethod(Invocation invocation) → dynamic: Invoked when a nonexistent method or property is accessed.
inherited
textToTokens(String text) → List<String>: Converts text to tokens.
tokenize(String text, {bool bos = false, bool eos = false}) → List<int>: Tokenizes the text.
tokensToText(List<String> tokens) → String: Converts tokens to text.
toString() → String: A string representation of this object.
inherited
vocab() → List<String>: Returns the vocabulary tokens.