reFtsTextToken top-level property
A regexp matching a standard FTS token.
It is defined as a sequence of characters from the L, N, and Co Unicode categories, additionally accepting Mn and Mc categories after the first position. This definition corresponds to the default behavior of the "unicode61" tokenizer, except for the aforementioned additional acceptance. The ftsText preprocessing function uses this regexp to identify standard FTS tokens.
See also: ftsTokenCategories and reFtsTermToken.
Implementation
final RegExp reFtsTextToken = RegExp(
"[\\p{L}\\p{N}\\p{Co}]"
"[\\p{L}\\p{N}\\p{Co}\\p{Mn}\\p{Mc}]*",
unicode: true,
);