embedding_tokenizer library

Public, native-only export of the Gemma SentencePiece EmbeddingTokenizer adapter (loadGemmaSentencePieceEmbeddingTokenizer, loadEmbeddingTokenizer, encodeForEmbedding) for native-only leaves that need to import it from a file that is itself native-only (never reached on web) — engine packages' embedding backends (e.g. flutter_edge_ai_litertlm's LiteRtEmbeddingBackend).

Prefer this over package:flutter_edge_ai_embeddings/src/embedding_tokenizer.dart in native-only files, matching the same pattern flutter_edge_ai_litertlm/lib/litert_bindings.dart uses for its FFI bindings: a stable public entry point instead of an implementation_imports lint on a src/ path.

This library is unconditional — importing it from code that is also reachable on web will fail to compile there (it pulls in dart_sentencepiece_tokenizer, which imports dart:io/dart:isolate unconditionally). It is deliberately NOT re-exported from this package's main barrel (flutter_edge_ai_embeddings.dart) for exactly that reason — see that barrel's module doc.

Constants

bosId → const int

Gemma special-token IDs, added by this package rather than by the loader: loadEmbeddingTokenizer passes an explicit SentencePieceConfig with addBosToken/addEosToken false, and every loader version takes the caller's config over the file's post_processor. So these are the only terminators encodeForEmbedding produces, whatever the file declares.

eosId → const int

siglipEosId → const int

SigLIP2's trailing end-of-sequence id. No BOS is prepended.

siglipPadId → const int

The id every position past the content is filled with — <pad>, id 0 in the SigLIP 2 vocabulary. See siglipSeqLen for why this one is not cosmetic: with right-padding it is the position the head actually pools.

siglipSeqLen → const int

SigLIP2's fixed context width and pad id, both taken from the model's own tokenizer.json, which bakes them in:

Functions

encodeForEmbedding(SentencePieceTokenizer tokenizer, String prefix, String text) → List<int>

Tokenizes (prefix + text) with Gemma BOS/EOS: [bosId, ...ids(prefix + text), eosId], where the ids are what SentencePiece produces for a tokenizer from loadEmbeddingTokenizer (its user-defined pieces kept whole) and tokenizer.encode(...).ids for any other.

encodeForSiglipEmbedding(SentencePieceTokenizer tokenizer, String text) → List<int>

Tokenizes text with SigLIP2's convention and returns exactly siglipSeqLen ids: no BOS, the lowercased content, one trailing siglipEosId, then right-padding with siglipPadId.

loadEmbeddingTokenizer(String tokenizerPath) → Future<SentencePieceTokenizer>

Loads the SentencePiece tokenizer at tokenizerPath — a .json (via TokenizerJsonLoader) or a raw SentencePiece .model file.

loadGemmaSentencePieceEmbeddingTokenizer(String tokenizerPath) → Future<EmbeddingTokenizer>

EmbeddingTokenizerFactory tear-off (design D-T1) — a thin EmbeddingTokenizer adapter over the loader and encodeForEmbedding: the same ids loadEmbeddingTokenizer + encodeForEmbedding give, but for a tokenizer.json only the tokenizer the embedding needs is built (see loadEmbeddingTokenizer). Same BOS=2/EOS=1 convention, same prefix + text concatenation order as the pre-generalization LiteRT path. Always returns attentionMask: null, tokenTypeIds: null — Gemma SentencePiece has no notion of either; the embedding pass pads internally and has no mask to report back.

loadSiglipSentencePieceEmbeddingTokenizer(String tokenizerPath) → Future<EmbeddingTokenizer>

EmbeddingTokenizerFactory tear-off for the SigLIP2 text tower — the same loader (.json/.model branch, embedding route) as Gemma's factory, but wrapped with SigLIP's no-BOS/single-EOS/lowercase convention instead.