EmbeddingTokenizerFactory typedef

EmbeddingTokenizerFactory = Future<EmbeddingTokenizer> Function(String tokenizerPath)

Factory that builds an EmbeddingTokenizer for a given on-disk tokenizer path.

MUST be a top-level or static function — a genuine tear-off — and never a closure or bound instance method, for the reason documented on EmbeddingPassDescriptor: on native the descriptor, and the pass config carrying this factory, are sent into core's freshly spawned worker isolate (see test/embedding/embedding_host_test.dart for the boundary), and only a top-level/static tear-off survives that trip intact — a closure over local/instance state is either rejected outright or silently loses the captured object identity, and a runtime registry populated via some register() call is empty in the freshly spawned isolate no matter what the main isolate registered into it before spawning.

Implementation

typedef EmbeddingTokenizerFactory =
    Future<EmbeddingTokenizer> Function(String tokenizerPath);