LoadOptions class final

How a model is loaded. The defaults are the settings the package's measurements used: 4096 tokens of context, flash attention on, KV cache f16, every layer on the GPU.

Annotations

• @immutable

Constructors

LoadOptions({int contextTokens = 4096, int? threads, int? gpuLayers, int? microBatchSize, int? batchSize, bool flashAttention = true, KvCacheQuant kvCacheK = KvCacheQuant.f16, KvCacheQuant kvCacheV = KvCacheQuant.f16, ComputeBackend compute = ComputeBackend.auto})

const

Properties

batchSize → int?

Logical batch size (n_batch); null (or 0) keeps the runtime's default.

final

compute → ComputeBackend

The compute backend asked for.

final

contextTokens → int

Context size in tokens (n_ctx).

final

flashAttention → bool

Flash attention; the quantized KV cache types need it.

final

gpuLayers → int?

Layers offloaded to the GPU; null (or a negative value) offloads all of them.

final

hashCode → int

The hash code for this object.

no setter, override

kvCacheK → KvCacheQuant

Storage type of the KV cache's keys.

final

kvCacheV → KvCacheQuant

Storage type of the KV cache's values.

final

microBatchSize → int?

Prompt tokens per GPU pass (n_ubatch). Null means 128 on iOS, where a hot phone aborted 512-token passes, and the runtime's default (512) elsewhere; 0 is the runtime's default everywhere.

final

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

threads → int?

CPU threads for generation and for prompt processing; null (or 0) lets the runtime choose.

final

Methods

copyWith({int? contextTokens, int? threads, int? gpuLayers, int? microBatchSize, int? batchSize, bool? flashAttention, KvCacheQuant? kvCacheK, KvCacheQuant? kvCacheV, ComputeBackend? compute}) → LoadOptions

A copy with the given fields replaced. A null argument keeps the current value.

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

toJson() → Map<String, Object?>

Every field, enums by name, in declaration order.

toString() → String

A string representation of this object.

override

Operators

operator ==(Object other) → bool

The equality operator.

override