decodeSenseVoiceText function

String decodeSenseVoiceText(
  1. List<String> vocabulary,
  2. List<int> tokens
)

Joins SentencePiece tokens: ▁ starts a word. Special tokens, such as <unk> and <|en|>, are skipped.

Implementation

String decodeSenseVoiceText(List<String> vocabulary, List<int> tokens) {
  final out = StringBuffer();
  for (final id in tokens) {
    final token = vocabulary[id];
    if (token.startsWith('<') && token.endsWith('>')) continue;
    out.write(token.replaceAll('▁', ' '));
  }
  return out.toString().trim();
}