ftsTerm function

String ftsTerm(
  1. String string
)

Prepares a String for use as an extended FTS token.

This function replaces ftsTermJoiners, if present in the input string, with spaces, performs NFC normalization, and replaces inner sequences of characters that do not match reFtsTermToken with ftsTermJoiner. The resulting token can be used as an indivisible unit in strings assigned to FTS fields and in FTS queries. It is "extended" because it can contain characters from the P and Sm categories in addition to the standard token characters. Such tokens may be used as identifiers, tags, etc.

The input string should be free from invisible characters that interfere with tokens by either causing unwanted cuts (e.g., U+00AD SHY) or by changing equality (e.g., U+FE0F VS16). Additionally, such characters may block NFC composition.

See also: ftsText.

Implementation

String ftsTerm(String string) {
  var norm = nfc(string.replaceAll(ftsTermJoiner, " "));
  var tokens = reFtsTermToken.allMatches(norm).map((var m) => m.group(0)!);
  return tokens.join(ftsTermJoiner);
}