decode method

String decode(
  1. List<int> tokens, {
  2. bool allowMalformed = true,
})

Decodes a list of tokens into a string.

WARNING: the default behaviour of this function is lossy, since decoded bytes are not guaranteed to be valid UTF-8. You can control this behaviour using the allowMalformed parameter, for instance, setting allowMalformed = false.

Example:

final enc = getEncoding("gpt2");
enc.decode([31373, 995]); // 'hello world'

Implementation

String decode(List<int> tokens, {bool allowMalformed = true}) {
  return _coreBPE.decodeNative(tokens).asString(
        allowMalformed: allowMalformed,
      );
}