InferenceEngine class abstract interface

A local runtime behind one narrow surface. It renders nothing: prompts arrive rendered, and stop strings, grammar and preserved tokens come in GenerationOptions.

load, generate, tokenize and countTokens, saveState, loadState and dispose run one at a time, in call order. After dispose every call throws StateError.

Implementers

Properties

contextEpoch → int
Counts the calls that change the context: generate, saveState and loadState add 1 when called, before they return, whether they later run, fail or are cancelled. An unchanged value means the context still holds what the caller left there.
no setter
hashCode → int
The hash code for this object.
no setterinherited
info → EngineInfo?
The loaded model; null before load completes and after dispose.
no setter
isLoaded → bool
Whether a model is loaded and the engine is not disposed.
no setter
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
supportsStatePersistence → bool
Whether saveState and loadState work on this runtime.
no setter

Methods

countTokens(String text) → Future<int>
(await tokenize(text)).length.
dispose() → Future<void>
Frees the model. Queued generations complete as cancelled, other queued calls fail with StateError, and a running generation is cancelled. Calling it again does nothing more.
generate(String prompt, [GenerationOptions options = const GenerationOptions()]) → Generation
Queues a generation from the rendered prompt and returns its handle at once. A runtime failure, or no model loaded when its turn comes, fails the handle; only a call after dispose throws (StateError).
load(String modelPath, [LoadOptions options = const LoadOptions()]) → Future<EngineInfo>
Loads modelPath. Throws StateError while a model is loaded or loading, ModelLoadException when the runtime cannot load it. A model without a chat template loads (EngineInfo.chatTemplate is then null).
loadState(String path) → Future<int>
Restores a state written by saveState; returns the tokens it holds. Failures throw StateFileException.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
saveState(String path, String promptPrefix) → Future<SavedState>
Fills the context with exactly promptPrefix, samples nothing, and writes the state to path. A recurrent state cannot be rewound, so the state ends at the prefix's last token. Failures throw StateFileException.
tokenize(String text) → Future<List<int>>
The tokens of text, special tokens added as generate adds them (the vocabulary decides about BOS).
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited