decodeTokenBytes method

List<Uint8List> decodeTokenBytes(
  1. List<int> tokens
)

Decodes a list of tokens into a list of bytes.

Useful for visualising tokenisation.

Example:

final enc = Tiktoken.getEncoderForModel(OpenAiModel.gpt_4);
enc.decodeTokenBytes([31373, 995]);

Implementation

List<Uint8List> decodeTokenBytes(List<int> tokens) =>
    tokens.map((token) => decodeSingleTokenBytes(token)).toList();