encodeForSiglipEmbedding function

List<int> encodeForSiglipEmbedding(SentencePieceTokenizer tokenizer, String text)

Tokenizes text with SigLIP2's convention and returns exactly siglipSeqLen ids: no BOS, lowercased content truncated to siglipSeqLen - 1, one trailing siglipEosId, then right-padding with siglipPadId.

The EOS survives truncation — content of 63 tokens or more yields [...first 63, EOS] with no padding, matching the reference.

Takes the content text ALONE. The adapter deliberately does not hand a TaskType prefix through — see loadSiglipSentencePieceEmbeddingTokenizer.

Implementation

List<int> encodeForSiglipEmbedding(
  SentencePieceTokenizer tokenizer,
  String text,
) {
  // Truncate the CONTENT to leave room for the EOS, rather than appending it
  // and cutting it back off — the reference (`tokenizers` with
  // `enable_truncation(64)`, and DJL's `LONGEST_FIRST` default, which is what
  // the reference Android app runs) keeps `<eos>` at index 63 on a long input.
  // Appending first silently dropped it and cost cosine 0.9683 on inputs over
  // the width.
  // `loadEmbeddingTokenizer` turns the tokenizer's own padding and truncation
  // off, so this is bare content and the width rule below is the only one that
  // applies. Without that the file's `{Fixed: 64}` block comes back applied and
  // the EOS appended here would land past the pad run.
  final content = tokenizer.encode(text.toLowerCase()).ids;
  final ids = <int>[...content.take(siglipSeqLen - 1), siglipEosId];
  return [...ids, ...List<int>.filled(siglipSeqLen - ids.length, siglipPadId)];
}