tokenize abstract method

Future<List<int>> tokenize(
  1. PromptValue promptValue, {
  2. Options? options,
})

Tokenizes the given prompt using the encoding used by the language model.

  • promptValue The prompt to tokenize.

Implementation

Future<List<int>> tokenize(
  final PromptValue promptValue, {
  final Options? options,
});