DonutTokenizer.fromVocab constructor

DonutTokenizer.fromVocab(Map<String, int> vocab)

Create a simple tokenizer with just a vocabulary mapping.

Useful for loading from sentencepiece model files or when only the vocabulary is available.

Implementation

factory DonutTokenizer.fromVocab(Map<String, int> vocab) {
  return DonutTokenizer(
    vocab: vocab,
    merges: [],
  );
}