encodeForSiglipEmbedding function

List<int> encodeForSiglipEmbedding(SentencePieceTokenizer tokenizer, String text)

Tokenizes text with SigLIP2's convention and returns exactly siglipSeqLen ids: no BOS, the lowercased content, one trailing siglipEosId, then right-padding with siglipPadId.

Content longer than siglipSeqLen - 1 (63 tokens) throws EmbeddingInputTooLongException with maxInputTokens: 64 and actualTokens = content + EOS. It used to be cut to its first 63 tokens — a vector of a different text, with no error.

Takes the content text ALONE. The adapter deliberately does not hand a TaskType prefix through — see loadSiglipSentencePieceEmbeddingTokenizer.

Implementation

List<int> encodeForSiglipEmbedding(
  SentencePieceTokenizer tokenizer,
  String text,
) {
  // `loadEmbeddingTokenizer` turns the tokenizer's own padding and truncation
  // off, so this is bare content and the width rule below is the only one that
  // applies. Without that the file's `{Fixed: 64}` block comes back applied and
  // the EOS appended here would land past the pad run.
  final content = _contentIds(tokenizer, text.toLowerCase());
  // Prefix-free, so the EOS is the only id beside the content.
  final actualTokens = content.length + 1;
  if (actualTokens > siglipSeqLen) {
    throw EmbeddingInputTooLongException(
      'The input is $actualTokens tokens (EOS included); the SigLIP 2 text '
      'tower takes at most $siglipSeqLen. Split the text and embed the '
      'pieces.',
      maxInputTokens: siglipSeqLen,
      actualTokens: actualTokens,
    );
  }
  final ids = <int>[...content, siglipEosId];
  return [...ids, ...List<int>.filled(siglipSeqLen - ids.length, siglipPadId)];
}