decodeSingleTokenBytes method

Uint8List decodeSingleTokenBytes(
  1. int token
)

Decodes a token into bytes.

NOTE: this will decode all special tokens.

Throws TiktokenError if the token is not in the vocabulary.

Example:

final enc = getEncoding("gpt2");
enc.decodeSingleTokenBytes(31373); // [104, 101, 108, 108, 111]

Implementation

Uint8List decodeSingleTokenBytes(int token) {
  return _coreBPE.decodeSingleTokenBytes(token).bytes;
}