embedding_tokenizer library

Public, native-only export of the Gemma SentencePiece EmbeddingTokenizer adapter (loadGemmaSentencePieceEmbeddingTokenizer, loadEmbeddingTokenizer, encodeForEmbedding) for native-only leaves that need to import it from a file that is itself native-only (never reached on web) — engine packages' embedding backends (e.g. flutter_gemma_litertlm's LiteRtEmbeddingBackend).

Prefer this over package:flutter_gemma_embeddings/src/embedding_tokenizer.dart in native-only files, matching the same pattern flutter_gemma_litertlm/lib/litert_bindings.dart uses for its FFI bindings: a stable public entry point instead of an implementation_imports lint on a src/ path.

This library is unconditional — importing it from code that is also reachable on web will fail to compile there (it pulls in dart_sentencepiece_tokenizer, which imports dart:io/dart:isolate unconditionally). It is deliberately NOT re-exported from this package's main barrel (flutter_gemma_embeddings.dart) for exactly that reason — see that barrel's module doc.

Constants

bosId → const int

Gemma special-token IDs. dart_sentencepiece_tokenizer defaults to the swapped pair (bosId=1, eosId=2), so we add them manually.

eosId → const int

siglipEosId → const int

SigLIP2's trailing end-of-sequence id. No BOS is prepended.

siglipPadId → const int

The id every position past the content is filled with — <pad>, id 0 in the SigLIP 2 vocabulary. See siglipSeqLen for why this one is not cosmetic: with right-padding it is the position the head actually pools.

siglipSeqLen → const int

SigLIP2's fixed context width and pad id, both taken from the model's own tokenizer.json, which bakes them in:

Functions

encodeForEmbedding(SentencePieceTokenizer tokenizer, String prefix, String text) → List<int>

Tokenizes (prefix + text) with Gemma BOS/EOS: [bosId, ...encode(prefix + text).ids, eosId].

encodeForSiglipEmbedding(SentencePieceTokenizer tokenizer, String text) → List<int>

Tokenizes text with SigLIP2's convention and returns exactly siglipSeqLen ids: no BOS, lowercased content truncated to siglipSeqLen - 1, one trailing siglipEosId, then right-padding with siglipPadId.

loadEmbeddingTokenizer(String tokenizerPath) → Future<SentencePieceTokenizer>

Loads the SentencePiece tokenizer at tokenizerPath — a .json (via TokenizerJsonLoader) or a raw SentencePiece .model file, matching exactly the branch litert_embedding_core.dart used pre-refactor.

loadGemmaSentencePieceEmbeddingTokenizer(String tokenizerPath) → Future<EmbeddingTokenizer>

EmbeddingTokenizerFactory tear-off (design D-T1) — a thin EmbeddingTokenizer adapter over loadEmbeddingTokenizer + encodeForEmbedding. Byte-identical to the pre-generalization LiteRT path: same BOS=2/EOS=1 convention, same prefix + text concatenation order. Always returns attentionMask: null, tokenTypeIds: null — Gemma SentencePiece has no notion of either; the forward pass pads/truncates internally and has no mask to report back.

loadSiglipSentencePieceEmbeddingTokenizer(String tokenizerPath) → Future<EmbeddingTokenizer>

EmbeddingTokenizerFactory tear-off for the SigLIP2 text tower — reuses the same base loadEmbeddingTokenizer (.json/.model branch) as Gemma, but wraps it with SigLIP's no-BOS/single-EOS/lowercase convention instead.