Tokenizer typedef

Tokenizer = Future<Iterable<Token>> Function(SourceText source, {NGramRange? nGramRange, TokenizingStrategy? strategy, Zone? zone})

Type definition of a function that returns a collection of Token from the source text.

Optional parameter zone is the name of the zone in a document in which the term is located.

Implementation

typedef Tokenizer = Future<Iterable<Token>> Function(SourceText source,
    {NGramRange? nGramRange, TokenizingStrategy? strategy, Zone? zone});