tokenizer property

LanguageTokenizer? tokenizer
final

A custom tokenizer for languages without built-in support.

Implementation

final LanguageTokenizer? tokenizer;