embedding_tokenizer library

Public, native-only export of the Gemma SentencePiece EmbeddingTokenizer adapter (loadGemmaSentencePieceEmbeddingTokenizer, loadEmbeddingTokenizer, encodeForEmbedding) for native-only leaves that need to import it from a file that is itself native-only (never reached on web) — engine packages' embedding backends (e.g. flutter_gemma_litertlm's LiteRtEmbeddingBackend).

Prefer this over package:flutter_gemma_embeddings/src/embedding_tokenizer.dart in native-only files, matching the same pattern flutter_gemma_litertlm/lib/litert_bindings.dart uses for its FFI bindings: a stable public entry point instead of an implementation_imports lint on a src/ path.

This library is unconditional — importing it from code that is also reachable on web will fail to compile there (it pulls in dart_sentencepiece_tokenizer, which imports dart:io/dart:isolate unconditionally). It is deliberately NOT re-exported from this package's main barrel (flutter_gemma_embeddings.dart) for exactly that reason — see that barrel's module doc.

Constants

bosId → const int

Gemma special-token IDs. dart_sentencepiece_tokenizer defaults to the swapped pair (bosId=1, eosId=2), so we add them manually.

eosId → const int

Functions

encodeForEmbedding(SentencePieceTokenizer tokenizer, String prefix, String text) → List<int>

Tokenizes (prefix + text) with Gemma BOS/EOS: [bosId, ...encode(prefix + text).ids, eosId].

loadEmbeddingTokenizer(String tokenizerPath) → Future<SentencePieceTokenizer>

Loads the SentencePiece tokenizer at tokenizerPath — a .json (via TokenizerJsonLoader) or a raw SentencePiece .model file, matching exactly the branch litert_embedding_core.dart used pre-refactor.

loadGemmaSentencePieceEmbeddingTokenizer(String tokenizerPath) → Future<EmbeddingTokenizer>

EmbeddingTokenizerFactory tear-off (design D-T1) — a thin EmbeddingTokenizer adapter over loadEmbeddingTokenizer + encodeForEmbedding. Byte-identical to the pre-generalization LiteRT path: same BOS=2/EOS=1 convention, same prefix + text concatenation order. Always returns attentionMask: null, tokenTypeIds: null — Gemma SentencePiece has no notion of either; the forward pass pads/truncates internally and has no mask to report back.