TokenEmbeddingPass class abstract base

An EmbeddingPass for an engine that tokenizes in Dart. Text only.

Subclasses open, run and close their graph; this class owns the tokenizer, the task prefix, the length check and the finalisation, and nobody else may override them.

Inheritance

Constructors

TokenEmbeddingPass({required String tokenizerPath, required EmbeddingTokenizerFactory tokenizerFactory})

Properties

hashCode → int
The hash code for this object.
no setterinherited
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited

Methods

close() → Future<void>
Frees everything the pass holds. Called exactly once by core on every path after construction (load failure, close, host shutdown). Idempotent; must not throw for a pass that never loaded.
override
closeGraph() → Future<void>
Frees the graph. Idempotent. Also called when openGraph threw, to free whatever it had allocated by then; never called when openGraph never ran.
embed(EmbeddingInput input, TaskType taskType) → Future<List<double>>
tokenizer.encode(taskType.prefix, input.text!) — the tokenizer adapter decides what the prefix means (SigLIP drops it). Throws EmbeddingInputTooLongException when the ids — prefix, BOS and EOS included — are longer than the graph's maxInputTokens; otherwise runs runTokens and finalises the result with the effective mask (the one the graph echoed, else the tokenizer's).
override
load() → Future<EmbeddingPassInfo>
Builds the tokenizer (tokenizerFactory(tokenizerPath)), calls openGraph, and reports text, the graph's dimension and its maxInputTokens.
override
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
openGraph() → Future<TokenPassGraph>
Opens the graph. Called by load after the tokenizer is built.
runTokens(TokenizedInput input) → Future<ForwardResult>
One forward run over input. The ids are already length-checked; a static-shape graph pads here, and echoes the padded mask in ForwardResult.attentionMask.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited