Explicit tokenizer and part-of-speech tagger boundary for English.
Implementations must align EnglishPreprocessResult.controls to their
backend tokens and return tokens with EnglishTokenMetadata already
populated. Concatenating every token's text and whitespace must reproduce
EnglishPreprocessResult.text exactly. In particular, multi-token slash
controls set isHead on the first token and provide empty phonemes for
continuation tokens.
No spaCy tokenizer or tagger adapter is bundled by the pure-Dart package.
- Implemented types
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- info → BackendInfo
-
Stable backend identity included in diagnostics and fixture metadata.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
tokenize(
EnglishPreprocessResult input) → List< MisakiToken> -
Tokenizes and tags
input, applying its inline controls. -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited