tokenize method

@override Future<List<int>> tokenize(PromptValue promptValue, { LLMOptions? options, })

Tokenizes the given prompt using the encoding used by the language model.

• promptValue The prompt to tokenize.

Implementation

@override
Future<List<int>> tokenize(
  final PromptValue promptValue, {
  final LLMOptions? options,
}) async {
  return promptValue
      .toString()
      .split(' ')
      .map((final word) => word.hashCode)
      .toList(growable: false);
}