loadGemmaSentencePieceEmbeddingTokenizer function

Future<EmbeddingTokenizer> loadGemmaSentencePieceEmbeddingTokenizer(
  1. String tokenizerPath
)

EmbeddingTokenizerFactory tear-off (design D-T1) — a thin EmbeddingTokenizer adapter over the loader and encodeForEmbedding: the same ids loadEmbeddingTokenizer + encodeForEmbedding give, but for a tokenizer.json only the tokenizer the embedding needs is built (see loadEmbeddingTokenizer). Same BOS=2/EOS=1 convention, same prefix + text concatenation order as the pre-generalization LiteRT path. Always returns attentionMask: null, tokenTypeIds: null — Gemma SentencePiece has no notion of either; the embedding pass pads internally and has no mask to report back.

Implementation

Future<EmbeddingTokenizer> loadGemmaSentencePieceEmbeddingTokenizer(
  String tokenizerPath,
) async {
  final tokenizer = await _load(tokenizerPath, forEmbedding: true);
  return _GemmaSentencePieceEmbeddingTokenizer(tokenizer);
}