llama_context_params class final

NOTE: changing the default values of parameters marked as EXPERIMENTAL may cause crashes or incorrect results in certain configurations https://github.com/ggml-org/llama.cpp/pull/7544

Inheritance
Implemented types
Available extensions

Constructors

llama_context_params()

Properties

abort_callback ↔ ggml_abort_callback
Abort callback if it returns true, execution of llama_decode() will be aborted currently works only with CPU execution
getter/setter pair
abort_callback_data ↔ Pointer<Void>
getter/setter pair
address → Pointer<T>

Available on T, provided by the StructAddress extension

The memory address of the underlying data.
no setter
attention_type → llama_attention_type
no setter
attention_typeAsInt ↔ int
attention type to use for embeddings
getter/setter pair
cb_eval ↔ ggml_backend_sched_eval_callback
getter/setter pair
cb_eval_user_data ↔ Pointer<Void>
getter/setter pair
ctx_type → llama_context_type
no setter
ctx_typeAsInt ↔ int
set the context type (e.g. MTP)
getter/setter pair
defrag_thold ↔ double
DEPRECATED defragment the KV cache if holes/size > thold, <= 0 disabled (default)
getter/setter pair
embeddings ↔ bool
if true, extract embeddings (together with logits)
getter/setter pair
flash_attn_type → llama_flash_attn_type
no setter
flash_attn_typeAsInt ↔ int
when to enable Flash Attention
getter/setter pair
hashCode → int
The hash code for this object.
no setterinherited
kv_unified ↔ bool
use a unified buffer across the input sequences when computing the attention try to disable when n_seq_max > 1 for improved performance when the sequences do not share a large prefix ref: https://github.com/ggml-org/llama.cpp/pull/14363
getter/setter pair
n_batch ↔ int
logical maximum batch size that can be submitted to llama_decode
getter/setter pair
n_ctx ↔ int
text context, 0 = from model
getter/setter pair
n_rs_seq ↔ int
number of recurrent-state snapshots per seq for rollback (0 = no rollback) EXPERIMENTAL
getter/setter pair
n_samplers ↔ int
getter/setter pair
n_seq_max ↔ int
max number of sequences (i.e. distinct states for recurrent models)
getter/setter pair
n_threads ↔ int
number of threads to use for generation
getter/setter pair
n_threads_batch ↔ int
number of threads to use for batch processing
getter/setter pair
n_ubatch ↔ int
physical maximum batch size
getter/setter pair
no_perf ↔ bool
measure performance timings
getter/setter pair
offload_kqv ↔ bool
offload the KQV ops (including the KV cache) to GPU
getter/setter pair
op_offload ↔ bool
offload host tensor operations to device
getter/setter pair
pooling_type → llama_pooling_type
no setter
pooling_typeAsInt ↔ int
whether to pool (sum) embedding results by sequence id
getter/setter pair
rope_freq_base ↔ double
RoPE base frequency, 0 = from model
getter/setter pair
rope_freq_scale ↔ double
RoPE frequency scaling factor, 0 = from model
getter/setter pair
rope_scaling_type → llama_rope_scaling_type
no setter
rope_scaling_typeAsInt ↔ int
RoPE scaling type, from enum llama_rope_scaling_type
getter/setter pair
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
samplers ↔ Pointer<llama_sampler_seq_config>
EXPERIMENTAL backend sampler chain configuration (make sure the caller keeps the sampler chains alive) note: the samplers must be sampler chains (i.e. use llama_sampler_chain_init)
getter/setter pair
swa_full ↔ bool
use full-size SWA cache (https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055) NOTE: setting to false when n_seq_max > 1 can cause bad performance in some cases ref: https://github.com/ggml-org/llama.cpp/pull/13845#issuecomment-2924800573
getter/setter pair
type_k → ggml_type
no setter
type_kAsInt ↔ int
data type for K cache EXPERIMENTAL
getter/setter pair
type_v → ggml_type
no setter
type_vAsInt ↔ int
data type for V cache EXPERIMENTAL
getter/setter pair
yarn_attn_factor ↔ double
YaRN magnitude scaling factor
getter/setter pair
yarn_beta_fast ↔ double
YaRN low correction dim
getter/setter pair
yarn_beta_slow ↔ double
YaRN high correction dim
getter/setter pair
yarn_ext_factor ↔ double
YaRN extrapolation mix factor, negative = from model
getter/setter pair
yarn_orig_ctx ↔ int
YaRN original context size
getter/setter pair

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited