decodeSingleTokenBytes method

Uint8List decodeSingleTokenBytes(
  1. int token
)

Decodes a token into bytes.

NOTE: this will decode all special tokens.

Throws TiktokenError if the token is not in the vocabulary.

Example:

final enc = Tiktoken.getEncoderForModel(OpenAiModel.gpt_4);
enc.decodeSingleTokenBytes(31373);

Implementation

Uint8List decodeSingleTokenBytes(int token) =>
    _coreBPE.decodeSingleTokenBytes(token).bytes;