decodeTokenBytes method

List<Uint8List> decodeTokenBytes(List<int> tokens)

Decodes a list of tokens into a list of bytes.

Useful for visualising tokenisation.

Example:

final enc = getEncoding("gpt2");
enc.decodeTokenBytes([31373, 995]); // [[104, 101, 108, 108, 111], [32, 119, 111, 114, 108, 100]]

Implementation

List<Uint8List> decodeTokenBytes(List<int> tokens) {
  return tokens.map((token) => decodeSingleTokenBytes(token)).toList();
}