decode method

String decode(List<int> tokens, { bool allowMalformed = true, })

Decodes a list of tokens into a string.

WARNING: the default behaviour of this function is lossy, since decoded bytes are not guaranteed to be valid UTF-8. You can control this behaviour using the allowMalformed parameter, for instance, setting allowMalformed = false.

Example:

final enc = Tiktoken.getEncoderForModel(OpenAiModel.gpt_4);
enc.decode([31373, 995]);

Implementation

String decode(List<int> tokens, {bool allowMalformed = true}) =>
    _coreBPE.decodeNative(tokens).asString(allowMalformed: allowMalformed);