loadTokenizerFromFile method

void loadTokenizerFromFile(
  1. String path
)

Load a tokenizer from a HuggingFace tokenizer.json file.

Implementation

void loadTokenizerFromFile(String path) {
  tokenizer = DonutTokenizer.fromFile(path);
  decoder.tokenizer = tokenizer;
}