loadEmbeddingTokenizer function

Future<SentencePieceTokenizer> loadEmbeddingTokenizer(
  1. String tokenizerPath
)

Loads the SentencePiece tokenizer at tokenizerPath — a .json (via TokenizerJsonLoader) or a raw SentencePiece .model file, matching exactly the branch litert_embedding_core.dart used pre-refactor.

Implementation

Future<SentencePieceTokenizer> loadEmbeddingTokenizer(
  String tokenizerPath,
) async {
  if (tokenizerPath.endsWith('.json')) {
    return TokenizerJsonLoader.fromJsonFile(
      tokenizerPath,
      config: const SentencePieceConfig(),
    );
  }
  return SentencePieceTokenizer.fromModelFile(
    tokenizerPath,
    config: const SentencePieceConfig(),
  );
}