decode method

String decode(
  1. List<int> tokens
)

Decode a list of token ids back to a string. Unpacks each merged id recursively down to raw byte ids, then UTF-8-decodes.

Implementation

String decode(List<int> tokens) {
  final bytes = <int>[];
  for (final t in tokens) {
    _expandTo(bytes, t);
  }
  return utf8.decode(bytes, allowMalformed: true);
}