tokenize abstract method
Tokenizes the given prompt using the encoding used by the language model.
promptValue
The prompt to tokenize.
Implementation
Future<List<int>> tokenize(
final PromptValue promptValue, {
final Options? options,
});
Tokenizes the given prompt using the encoding used by the language model.
promptValue
The prompt to tokenize.Future<List<int>> tokenize(
final PromptValue promptValue, {
final Options? options,
});