tokenizerExportScript constant

String const tokenizerExportScript

Python script to export tokenizer in HuggingFace format.

Implementation

static const String tokenizerExportScript = '''
from transformers import DonutProcessor

processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
processor.tokenizer.save_pretrained("./tokenizer_output/")
# This creates tokenizer.json which can be loaded by DonutTokenizer.fromFile()
''';