tokenize function

List<String> tokenize(
  1. String text
)

Splits text into word tokens, individual punctuation marks, and underscores.

Implementation

List<String> tokenize(String text) {
  return _tokenRe.allMatches(text).map((m) => m.group(0)!).toList();
}