tokenize abstract method

Future<List<int>> tokenize(
  1. String text
)

The model's token ids for text, as its tokenizer splits it — control tokens written as text (<end_of_turn>) included — with no BOS added.

Implementation

Future<List<int>> tokenize(String text);