decode method

String decode(List<int> ids, { bool skipSpecialTokens = true, })

Decode a list of token IDs back to text.

ids: list of token IDs skipSpecialTokens: if true, removes BOS/EOS/PAD tokens

Implementation

String decode(List<int> ids, {bool skipSpecialTokens = true}) {
  final tokens = <String>[];
  for (final id in ids) {
    final token = idToToken[id];
    if (token == null) continue;

    if (skipSpecialTokens) {
      if (token == bosToken || token == eosToken || token == padToken) {
        continue;
      }
    }

    tokens.add(token);
  }

  // Join tokens and handle SentencePiece space markers
  var text = tokens.join('');
  text = text.replaceAll('▁', ' ');

  // Clean up leading space
  if (text.startsWith(' ')) {
    text = text.substring(1);
  }

  return text;
}