core/image_tokenizer library

Classes

ImageTokenizer

Handles proper image tokenization for multimodal AI models to prevent "Prompt contained 0 image tokens but received 1 images" errors and corruption that causes repeating text patterns.

Enums

ModelType

Model types that require different tokenization approaches

Exceptions / Errors

ImageTokenizationException

Exception thrown when image tokenization fails