loadEmbeddingTokenizer function

Future<SentencePieceTokenizer> loadEmbeddingTokenizer(String tokenizerPath)

Loads the SentencePiece tokenizer at tokenizerPath — a .json (via TokenizerJsonLoader) or a raw SentencePiece .model file.

The returned tokenizer has any padding and truncation the file declares DISABLED: encode() hands back bare content, never a fixed-width row. Width and terminators are the caller's — encodeForEmbedding, encodeForSiglipEmbedding, and the forward pass that owns seqLen. (A raw .model carries no such blocks, so there the guarantee is free.)

Throws StateError if the resolved dart_sentencepiece_tokenizer accepts the calls that disable those settings and leaves them set anyway. That is a check on the tokenizer's config, not a guarantee about encode()'s output.

Implementation

Future<SentencePieceTokenizer> loadEmbeddingTokenizer(
  String tokenizerPath,
) async {
  // Why the disable lives here and not in each profile.
  //
  // On 1.4.1 a SigLIP 2 `tokenizer.json` (which declares `{Fixed: 64, Right}`)
  // comes back already padded: the loader applies the file's own blocks
  // regardless of the config passed here. Both `encodeForEmbedding` and
  // `encodeForSiglipEmbedding` append their own EOS after whatever they are
  // handed, which puts it past the pad run — every id in range, nothing thrown,
  // and a different vector, since SigLIP pools the LAST position.
  //
  // Undoing that afterwards was possible but was a heuristic over a pipeline
  // nobody checks: it assumed right-padding with id 0, and a file declaring
  // `"direction": "Left"` or another `pad_id` would have slipped through
  // silently. Turning the feature off is the same result without the assumption,
  // and it keeps this package owning its own width rule.
  //
  // `_applyTokenizerSettings` is HF-JSON-only, so on the `.model` arm the two
  // calls are a no-op today. They stay because the contract above is one
  // sentence for both arms, and it should not quietly depend on which ran.
  final tokenizer = tokenizerPath.endsWith('.json')
      ? await TokenizerJsonLoader.fromJsonFile(
          tokenizerPath,
          config: const SentencePieceConfig(),
        )
      : await SentencePieceTokenizer.fromModelFile(
          tokenizerPath,
          config: const SentencePieceConfig(),
        );
  tokenizer
    ..noPadding()
    ..noTruncation();

  // The cascade above is the mechanism; `requireBareContent` is the
  // enforcement, and it is separate because `^1.4.1` is open-topped. See its
  // doc for exactly which failure shapes it does and does not cover — the ones
  // it cannot see are `siglip_loader_contract_test.dart`'s job.
  return requireBareContent(tokenizer, tokenizerPath);
}