loadSiglipSentencePieceEmbeddingTokenizer function

Future<EmbeddingTokenizer> loadSiglipSentencePieceEmbeddingTokenizer(String tokenizerPath)

EmbeddingTokenizerFactory tear-off for the SigLIP2 text tower — reuses the same base loadEmbeddingTokenizer (.json/.model branch) as Gemma, but wraps it with SigLIP's no-BOS/single-EOS/lowercase convention instead.

The returned tokenizer ignores the TaskType prefix. TaskType is an EmbeddingGemma convention ('task: search result | query: '), and SigLIP's text tower has no vocabulary for it: a prefix would be embedded as literal leading text and move the vector off the space it shares with the vision tower, which encodes an image with no prefix at all. Dropping it is not a silent liberty — CommonEmbeddingModel.generateEmbedding DEFAULTS to TaskType.retrievalQuery, so rejecting a non-empty prefix would make this profile unreachable through the only public API, and honoring one would make retrievalQuery and retrievalDocument of the same string two different points.

Implementation

Future<EmbeddingTokenizer> loadSiglipSentencePieceEmbeddingTokenizer(
  String tokenizerPath,
) async {
  final tokenizer = await loadEmbeddingTokenizer(tokenizerPath);
  _assertSiglip2Vocab(tokenizer, tokenizerPath);
  return _SiglipSentencePieceEmbeddingTokenizer(tokenizer);
}