LlmInferenceEngine class

Utility to query an LLM with a prompt and receive its response as a stream.

Inheritance

Constructors

LlmInferenceEngine(LlmInferenceOptions _options, {Duration timeout = const Duration(seconds: 10), int maxRetries = 2}): Utility to query an LLM with a prompt and receive its response as a stream.

hashCode → int: The hash code for this object.
no setterinherited
maxRetries → int: Number of times to invoke the timeout loop before giving up.
final
runtimeType → Type: A representation of the runtime type of the object.
no setterinherited
timeout → Duration: Length of time the Engine is willing to wait for a response from generateResponse before assuming that the inference task has failed and should be resumed.
final

cancel() → void: Terminates an in-progress query, closing down the stream.
dispose() → void: Releases all native resources and closes any open streams.
generateResponse(String text) → Stream<String>: Generates a response based on the input text.
override
noSuchMethod(Invocation invocation) → dynamic: Invoked when a nonexistent method or property is accessed.
inherited
restart(void callback()) → void: Powers down the executor and isolate and then powers up another one and invokes the callback once that replacement executor is awake.
sizeInTokens(String text) → Future<int>: Runs an invocation of only the tokenization for the LLM, and returns the size (in tokens) of the result.
override
toString() → String: A string representation of this object.
inherited