SignWritingTokenizer class
SignWritingTokenizer class extends BaseTokenizer for tokenization and detokenization of SignWriting symbols.
- Inheritance
-
- Object
- BaseTokenizer
- SignWritingTokenizer
Constructors
- SignWritingTokenizer({int? startingIndex})
- Constructor for SignWritingTokenizer.
Properties
- bosToken ↔ String
-
getter/setter pairinherited
- bosTokenId ↔ int
-
getter/setter pairinherited
- eosToken ↔ String
-
getter/setter pairinherited
- eosTokenId ↔ int
-
getter/setter pairinherited
- hashCode → int
-
The hash code for this object.
no setterinherited
-
i2s
↔ Map<
int, String> -
getter/setter pairinherited
- length → int
-
no setterinherited
- padToken ↔ String
-
getter/setter pairinherited
- padTokenId ↔ int
-
getter/setter pairinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
-
s2i
↔ Map<
String, int> -
getter/setter pairinherited
- unkToken ↔ String
-
getter/setter pairinherited
- unkTokenId ↔ int
-
getter/setter pairinherited
Methods
-
detokenize(
List< int> tokens) → String -
Detokenizes the tokens.
inherited
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
textToTokens(
String text, {bool boxPosition = false}) → List< String> -
Converts text to tokens.
override
-
tokenize(
String text, {bool bos = false, bool eos = false}) → List< int> -
Tokenizes the text.
inherited
-
tokensToText(
List< String> tokens) → String -
Converts tokens to text.
override
-
toString(
) → String -
A string representation of this object.
inherited
-
vocab(
) → List< String> -
Returns the vocabulary tokens.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited
Static Methods
-
tokenizeSymbol(
SignSymbol symbol, {bool boxPosition = false}) → List< String> - Method to tokenize a SignSymbol into a list of tokens.
-
tokens(
) → List< String> - Method to generate the list of tokens for SignWriting symbols.