ftsWords function

Iterable<String> ftsWords(
  1. String string
)

Extracts all standard FTS tokens from a String.

This function NFC-normalizes the input string and returns all character sequences that match the reFtsTextToken regexp. See ftsText for details on how invisible characters in string interfere with tokenization.

Implementation

Iterable<String> ftsWords(String string) {
  var norm = nfc(string);
  return reFtsTextToken.allMatches(norm).map((var m) => m.group(0)!);
}