LlmInferenceOptions class
Configuration object for a MediaPipe text classifier.
See also:
This io-friendly implementation is not immutable to track whether the native memory has been created and ultimately released. All values used by pkg:equatable are in fact immutable.
- Inheritance
-
- Object
- BaseLlmInferenceOptions
- LlmInferenceOptions
Constructors
- LlmInferenceOptions.cpu({required String modelPath, required String cacheDir, required int maxTokens, required double temperature, required int topK, int? randomSeed})
- LlmInferenceOptions.gpu({required String modelPath, required int sequenceBatchSize, required int maxTokens, required double temperature, required int topK, int decodeStepsPerSync = 3, int? randomSeed})
Properties
- cacheDir → String
-
Directory path for storing model related tokenizer and cache weights. The
user is responsible for providing the directory that can be writable by the
program. Used by CPU only.
final
- decodeStepsPerSync → int
-
Number of decode steps per sync. Used by GPU only. The default value is 3.
final
- hashCode → int
-
The hash code for this object.
no setterinherited
- isClosed → bool
-
Tracks whether dispose has been called.
no setter
- loraPath → String
-
Path to the LoRA tflite flatbuffer file. Optional (default is empty string).
This is only compatible with GPU models.
final
- maxTokens → int
-
The total length of the kv-cache.
final
- modelPath → String
-
The path that points to the tflite model file to use for inference.
final
-
props
→ List<
Object?> -
The list of properties that will be used to determine whether
two instances are equal.
no setterinherited
- randomSeed → int
-
Random seed for sampling tokens.
final
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- sequenceBatchSize → int
-
Sequence batch size for encoding. Used by GPU only. Number of input tokens
to process at a time for batch processing. Setting this value to 1 means
both the encoding and decoding share the same graph of sequence length
of 1. Setting this value to 0 means the batch size will be optimized
programmatically.
final
- stringify → bool?
-
If set to
true
, thetoString
method will be overridden to output this instance'sprops
.no setterinherited - temperature → double
-
Randomness when decoding the next token.
final
- topK → int
-
Top K number of tokens to be sampled from for each decoding step.
final
Methods
-
copyToNative(
) → Pointer< LlmSessionConfig> - Copies this options object into native memory for use by an engine.
-
dispose(
) → void - Releases the native memory behind this options object.
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited