loadGemmaSentencePieceEmbeddingTokenizer function

Future<EmbeddingTokenizer> loadGemmaSentencePieceEmbeddingTokenizer(
  1. String tokenizerPath
)

EmbeddingTokenizerFactory tear-off (design D-T1) — a thin EmbeddingTokenizer adapter over loadEmbeddingTokenizer + encodeForEmbedding. Byte-identical to the pre-generalization LiteRT path: same BOS=2/EOS=1 convention, same prefix + text concatenation order. Always returns attentionMask: null, tokenTypeIds: null — Gemma SentencePiece has no notion of either; the forward pass pads/truncates internally and has no mask to report back.

Implementation

Future<EmbeddingTokenizer> loadGemmaSentencePieceEmbeddingTokenizer(
  String tokenizerPath,
) async {
  final tokenizer = await loadEmbeddingTokenizer(tokenizerPath);
  return _GemmaSentencePieceEmbeddingTokenizer(tokenizer);
}