tokenize abstract method

List<MisakiToken> tokenize(
  1. EnglishPreprocessResult input
)

Tokenizes and tags input, applying its inline controls.

Implementation

List<MisakiToken> tokenize(EnglishPreprocessResult input);