ftsTokenCategories top-level constant
String
const ftsTokenCategories
Unicode categories allowed as part of an SQLite FTS token.
This library configures the "unicode61" tokenizer of FTS tables to consider
these Unicode categories as token characters (extending the default L,
N, and Co categories with P, Sm, Mn, and Mc). This means, for
example, that the text universal, state-of-the-art AC/DC adapter assigned
to an FTS field will be tokenized into universal,, state-of-the-art,
AC/DC, and adapter tokens. In order for the tokenizer to produce a set
of tokens similar to the default behavior, the text must be preprocessed
(e.g., via the ftsText function) before being assigned.
See also: reFtsTextToken, reFtsTermToken, and SearchIndex.new.
Implementation
const String ftsTokenCategories = "L* N* Co P* Sm Mn Mc";