tokenCategories top-level constant

String const tokenCategories

Unicode categories allowed as part of an SQLite FTS token.

This library configures the "unicode61" tokenizer of FTS tables to consider these Unicode categories as token characters (extending the default L, N, and Co with P and S). This means that, e.g., the text universal, state-of-the-art AC/DC adapter assigned to an FTS field will be tokenized into universal,, state-of-the-art, AC/DC, and adapter tokens. In order for the tokenizer to produce the same set of tokens as the default behavior, the text must be preprocessed (e.g., via the ftsText function) before being assigned.

See also: reFtsTextToken and reFtsTermToken.

Implementation

const String tokenCategories = "L* N* Co P* S*";