Tokens constructor

Tokens({
  1. double termDecay = 0.1,
})

Construct tokens

see: TermMapping.termDecay for explanation of termDecay usage.

Implementation

Tokens({double termDecay = 0.1})
    : super(
          (String str, bool caseSensitive, bool unicode) =>
              LinkedHashSet<String>.from((caseSensitive
                      ? str
                      : str.toLowerCase())
                  .split(
                      unicode ? _matchSeparatorUnicode : _matchSeparatorAscii)
                  .where((final term) => term.isNotEmpty)),
          termDecay);