LlmInferenceEngine class
Utility to query an LLM with a prompt and receive its response as a stream.
- Inheritance
-
- Object
- BaseLlmInferenceEngine
- LlmInferenceEngine
Constructors
- LlmInferenceEngine(LlmInferenceOptions _options, {Duration timeout = const Duration(seconds: 10), int maxRetries = 2})
- Utility to query an LLM with a prompt and receive its response as a stream.
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- maxRetries → int
-
Number of times to invoke the timeout loop before giving up.
final
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- timeout → Duration
-
Length of time the Engine is willing to wait for a response from
generateResponse
before assuming that the inference task has failed and should be resumed.final
Methods
-
cancel(
) → void - Terminates an in-progress query, closing down the stream.
-
dispose(
) → void - Releases all native resources and closes any open streams.
-
generateResponse(
String text) → Stream< String> -
Generates a response based on the input text.
override
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
restart(
void callback()) → void - Powers down the executor and isolate and then powers up another one and invokes the callback once that replacement executor is awake.
-
sizeInTokens(
String text) → Future< int> -
Runs an invocation of only the tokenization for the LLM, and returns the
size (in tokens) of the result.
override
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited