parseSenseVoiceTokens function

List<String> parseSenseVoiceTokens(
  1. String text
)

Parses tokens.txt: one token and its ID per line, IDs 0, 1, 2, ...

Implementation

List<String> parseSenseVoiceTokens(String text) {
  final tokens = <String>[];
  for (final line in text.split('\n')) {
    if (line.isEmpty) continue;
    final space = line.lastIndexOf(' ');
    if (space <= 0 ||
        int.tryParse(line.substring(space + 1)) != tokens.length) {
      throw FormatException('tokens.txt: line ${tokens.length + 1} is "$line"');
    }
    tokens.add(line.substring(0, space));
  }
  return tokens;
}