allTokens static method

List<String> allTokens(
  1. String s
)

Every word of s (normalised, canonical), filler included.

Implementation

static List<String> allTokens(String s) => text_norm
    .normaliseForMatch(s)
    .split(' ')
    .map(_canon)
    .where((t) => t.isNotEmpty)
    .toList(growable: false);