addTaskTokens method

void addTaskTokens(List<Map<String, dynamic>> labels)

Add task-specific special tokens for a dataset.

This should be called before inference to ensure all task-specific tokens are in the vocabulary.

labels: list of JSON objects representing ground truth labels. Each JSON object's keys define the special tokens needed.

Implementation

void addTaskTokens(List<Map<String, dynamic>> labels) {
  if (tokenizer == null) return;

  final tokens = <String>{};
  for (final label in labels) {
    _extractSpecialTokensFromJson(label, tokens);
  }

  tokenizer!.addSpecialTokens(tokens.toList());
}