encodeSingleToken method

int encodeSingleToken(List<int> bytes)

Encodes text corresponding to a single token to its token value.

NOTE: this will encode all special tokens.

Throws TiktokenError if the token is not in the vocabulary.

Example:

final enc = Tiktoken.getEncoderForModel(OpenAiModel.gpt_4);
enc.encodeSingleToken("hello")

Implementation

int encodeSingleToken(List<int> bytes) {
  return _coreBPE.encodeSingleToken(ByteArray.fromList(bytes));
}