ftsTokenCategories top-level constant

String const ftsTokenCategories

Unicode categories allowed as part of an SQLite FTS token.

This library configures the "unicode61" tokenizer of FTS tables to consider these Unicode categories as token characters (extending the default L, N, and Co categories with P, Sm, Mn, and Mc). This means, for example, that the text universal, state-of-the-art AC/DC adapter assigned to an FTS field will be tokenized into universal,, state-of-the-art, AC/DC, and adapter tokens. In order for the tokenizer to produce a set of tokens similar to the default behavior, the text must be preprocessed (e.g., via the ftsText function) before being assigned.

See also: reFtsTextToken, reFtsTermToken, and SearchIndex.new.

Implementation

const String ftsTokenCategories = "L* N* Co P* Sm Mn Mc";