encodeForEmbedding function

List<int> encodeForEmbedding(
  1. SentencePieceTokenizer tokenizer,
  2. String prefix,
  3. String text
)

Tokenizes (prefix + text) with Gemma BOS/EOS: [bosId, ...ids(prefix + text), eosId], where the ids are what SentencePiece produces for a tokenizer from loadEmbeddingTokenizer (its user-defined pieces kept whole) and tokenizer.encode(...).ids for any other.

Does NOT pad to a fixed sequence length, nor check one — that is the embedding pass's job (it owns seqLen, e.g. LiteRT's compiled input tensor width, and refuses a longer input).

Implementation

List<int> encodeForEmbedding(
  SentencePieceTokenizer tokenizer,
  String prefix,
  String text,
) => <int>[bosId, ..._contentIds(tokenizer, prefix + text), eosId];