loadSiglipSentencePieceEmbeddingTokenizer function

Future<EmbeddingTokenizer> loadSiglipSentencePieceEmbeddingTokenizer(String tokenizerPath)

EmbeddingTokenizerFactory tear-off for the SigLIP2 text tower — the same loader (.json/.model branch, embedding route) as Gemma's factory, but wrapped with SigLIP's no-BOS/single-EOS/lowercase convention instead.

The returned tokenizer ignores the TaskType prefix. TaskType is an EmbeddingGemma convention ('task: search result | query: '), and SigLIP's text tower has no vocabulary for it: a prefix would be embedded as literal leading text and move the vector off the space it shares with the vision tower, which encodes an image with no prefix at all. Dropping it is not a silent liberty — every EmbeddingModel.embed names a task, and only TaskType.none has an empty prefix, so rejecting a non-empty one would make this profile reachable only through none, and honoring one would make retrievalQuery and retrievalDocument of the same string two different points.

Implementation

Future<EmbeddingTokenizer> loadSiglipSentencePieceEmbeddingTokenizer(
  String tokenizerPath,
) async {
  final tokenizer = await _load(tokenizerPath, forEmbedding: true);
  _assertSiglip2Vocab(tokenizer, tokenizerPath);
  return _SiglipSentencePieceEmbeddingTokenizer(tokenizer);
}