encodeForEmbedding function

List<int> encodeForEmbedding(
  1. SentencePieceTokenizer tokenizer,
  2. String prefix,
  3. String text
)

Tokenizes (prefix + text) with Gemma BOS/EOS: [bosId, ...encode(prefix + text).ids, eosId].

Does NOT pad/truncate to a fixed sequence length — that is the forward pass's job (it owns seqLen, e.g. LiteRT's compiled input tensor width).

Implementation

List<int> encodeForEmbedding(
  SentencePieceTokenizer tokenizer,
  String prefix,
  String text,
) {
  final encoded = tokenizer.encode(prefix + text);
  return <int>[bosId, ...encoded.ids, eosId];
}