tokenize method

List<int> tokenize(
  1. String text, {
  2. bool addSpecial = true,
})

Tokenizes text with the native LiteRT-LM tokenizer.

Implementation

List<int> tokenize(String text, {bool addSpecial = true}) {
  throw UnsupportedError('LiteRT-LM runtime requires a native platform.');
}