addSpecialTokens method

void addSpecialTokens(List<String> tokens)

Add special tokens to the tokenizer vocabulary.

Implementation

void addSpecialTokens(List<String> tokens) {
  tokenizer?.addSpecialTokens(tokens);
}