loadEmbeddingTokenizer function

Future<SentencePieceTokenizer> loadEmbeddingTokenizer(String tokenizerPath)

Loads the SentencePiece tokenizer at tokenizerPath — a .json (via TokenizerJsonLoader) or a raw SentencePiece .model file.

The returned tokenizer has any padding and truncation the file declares DISABLED: encode() hands back bare content, never a fixed-width row. Width and terminators are the caller's — encodeForEmbedding, encodeForSiglipEmbedding, and the embedding pass that owns seqLen. (A raw .model carries no such blocks, so there the guarantee is free.)

The returned tokenizer is the dependency's, loaded from the file as it always was: its own encode() is unchanged. encodeForEmbedding and encodeForSiglipEmbedding add what this function read from the file, and tokenize as SentencePiece does: the model's user-defined pieces (Gemma's </div>, <mask>, <start_of_turn>, …) matched first and kept whole, and <pad>, <eos>, <bos> and <unk> never produced from text. For a HuggingFace tokenizer.json in Gemma's shape the text between two pieces needs a second tokenizer, loaded from the same file without its added_tokens. It is built the first time one of those two functions runs on the returned tokenizer, by reading tokenizerPath again: a StateError then if the file is gone or has changed since. (The embedding route — loadGemmaSentencePieceEmbeddingTokenizer, loadSiglipSentencePieceEmbeddingTokenizer — builds only that second tokenizer and never the first.)

Throws UnsupportedError for a BPE model with user-defined pieces whose normalizer this package cannot reproduce them under, and FormatException for a tokenizer.json whose added tokens the file alone cannot type (user_defined_pieces.dart says which) — both rather than return ids that differ from SentencePiece's without a sign.

Throws StateError if the resolved dart_sentencepiece_tokenizer accepts the calls that disable those settings and leaves them set anyway. That is a check on the tokenizer's config, not a guarantee about encode()'s output.

Implementation

Future<SentencePieceTokenizer> loadEmbeddingTokenizer(
  String tokenizerPath,
) async {
  // Why the disable lives here and not in each profile.
  //
  // On 1.4.1 a SigLIP 2 `tokenizer.json` (which declares `{Fixed: 64, Right}`)
  // comes back already padded: the loader applies the file's own blocks
  // regardless of the config passed here. Both `encodeForEmbedding` and
  // `encodeForSiglipEmbedding` append their own EOS after whatever they are
  // handed, which puts it past the pad run — every id in range, nothing thrown,
  // and a different vector, since SigLIP pools the LAST position.
  //
  // Undoing that afterwards was possible but was a heuristic over a pipeline
  // nobody checks: it assumed right-padding with id 0, and a file declaring
  // `"direction": "Left"` or another `pad_id` would have slipped through
  // silently. Turning the feature off is the same result without the assumption,
  // and it keeps this package owning its own width rule.
  //
  // `_applyTokenizerSettings` is HF-JSON-only, so on the `.model` arm the two
  // calls are a no-op today. They stay because the contract above is one
  // sentence for both arms, and it should not quietly depend on which ran.
  return _load(tokenizerPath, forEmbedding: false);
}