decode method
Decodes a list of tokens into a string.
WARNING: the default behaviour of this function is lossy, since decoded bytes are not
guaranteed to be valid UTF-8. You can control this behaviour using the allowMalformed
parameter,
for instance, setting allowMalformed = false
.
Example:
final enc = getEncoding("gpt2");
enc.decode([31373, 995]); // 'hello world'
Implementation
String decode(List<int> tokens, {bool allowMalformed = true}) {
return _coreBPE.decodeNative(tokens).asString(
allowMalformed: allowMalformed,
);
}