EnglishTextTokenizer class

A TextTokenizer implementation for the English languages.

Inheritance

• Object
• BaseTextTokenizer
• EnglishTextTokenizer

Constructors

EnglishTextTokenizer()

Properties

hashCode → int

The hash code for this object.

no setter, inherited

language → Languages

Gets the language for the tokenizer.

no setter, override

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

Methods

convertWord(String word) → String?

Converts a word into all lower case and checks if it is a known stop word. If it is, then the word will be discarded and will not be considered as a valid token.

inherited

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

stopWords() → Set<String>

Gets all stop-words for a language.

override

tokenize(String text) → Set<String>

Tokenize a text and discards all stop-words from it.

inherited

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited