Tokenizer class
Tokenizer for Kokoro TTS
This class handles text normalization, phoneme conversion, and tokenization using the malsami library (Dart port of Malsami G2P engine).
Constructors
- Tokenizer({TokenizerConfig? config})
- Creates a tokenizer with optional configuration
Properties
- config → TokenizerConfig?
-
The configuration for this tokenizer
final
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
debugPunctuationProcessing(
String text) → void - Debug method to analyze how G2P processes text with punctuation
-
ensureInitialized(
) → Future< void> - Ensures the tokenizer is initialized
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
phonemize(
String text, {String lang = 'en-us'}) → Future< String> - Converts text to phonemes, preserving all original spacing and punctuation.
-
phonemizeWithTokens(
String text, String lang) → Future< (String, List< MToken> )> - Converts text to phonemes and returns both phonemes and tokens
-
tokenize(
String phonemes) → List< int> - Tokenizes phonemes into token IDs
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited
Static Methods
-
normalizeText(
String text) → String - Normalizes input text