decodeSingleTokenBytes method
Decodes a token into bytes.
NOTE: this will decode all special tokens.
Throws TiktokenError
if the token is not in the vocabulary.
Example:
final enc = getEncoding("gpt2");
enc.decodeSingleTokenBytes(31373); // [104, 101, 108, 108, 111]
Implementation
Uint8List decodeSingleTokenBytes(int token) {
return _coreBPE.decodeSingleTokenBytes(token).bytes;
}