WordpieceTokenizer class

Constructors

WordpieceTokenizer({required Map<String, int> encoder, required List<String> decoder, required String unkString, required int startToken, required int endToken, required int unkToken, required int maxInputTokens, required int maxInputCharsPerWord})

Properties

decoder → List<String>
final
encoder → Map<String, int>
final
endToken → int
final
hashCode → int
The hash code for this object.
no setterinherited
maxInputCharsPerWord → int
final
maxInputTokens → int
final
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
startToken → int
final
unkString → String
final
unkToken → int
final

Methods

clearCaches() → void
detokenize(List<int> tokens) → String
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
tokenize(String text, {int? maxTokens}) → List<TextAndTokens>
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited