tokenize method

  1. @override
Future<List<int>> tokenize(
  1. PromptValue promptValue, {
  2. LLMOptions? options,
})
override

Tokenizes the given prompt using the encoding used by the language model.

  • promptValue The prompt to tokenize.

Implementation

@override
Future<List<int>> tokenize(
  final PromptValue promptValue, {
  final LLMOptions? options,
}) async {
  return promptValue
      .toString()
      .split(' ')
      .map((final word) => word.hashCode)
      .toList(growable: false);
}