lib_llama_cpp_ffi library

Classes

ggml_backend
ggml_backend_buffer
ggml_backend_buffer_type
ggml_backend_dev_caps
functionality supported by the device
ggml_backend_dev_props
all the device properties
ggml_backend_device
ggml_backend_event
ggml_backend_feature
Get a list of feature flags supported by the backend (returns a NULL-terminated array)
ggml_backend_graph_copy
Utils
ggml_backend_meta_split_state
ggml_backend_reg
ggml_backend_sched
ggml_bf16_t
google brain half-precision bfloat16
ggml_cgraph
ggml_context
ggml_cplan
the compute plan that needs to be prepared for ggml_graph_compute() since https://github.com/ggml-org/ggml/issues/287
ggml_gallocr
ggml_init_params
ggml_object
ggml_opt_context
ggml_opt_dataset
ggml_opt_optimizer_params
parameters that control which optimizer is used and how said optimizer tries to find the minimal loss
ggml_opt_params
parameters for initializing a new optimization context
ggml_opt_result
ggml_tallocr
Tensor allocator
ggml_tensor
n-dimensional tensor
ggml_threadpool
ggml_threadpool_params
threadpool params Use ggml_threadpool_params_default() or ggml_threadpool_params_init() to populate the defaults
ggml_type_traits
ggml_type_traits_cpu
gguf_context
lib_llama_cpp_chat_templates
lib_llama_cpp_media_blob
lib_llama_cpp_media_context
llama_adapter_lora
lora adapter
llama_batch
Input data for llama_encode/llama_decode A llama_batch object can contain input about one or many sequences The provided arrays (i.e. token, embd, pos, etc.) must have size of n_tokens
llama_chat_message
used in chat template
llama_context
llama_context_params
NOTE: changing the default values of parameters marked as EXPERIMENTAL may cause crashes or incorrect results in certain configurations https://github.com/ggml-org/llama.cpp/pull/7544
llama_logit_bias
llama_memory_i
llama_model
llama_model_imatrix_data
llama_model_kv_override
llama_model_params
llama_model_quantize_params
model quantization parameters
llama_model_tensor_buft_override
llama_model_tensor_override
llama_opt_params
llama_perf_context_data
Performance utils
llama_perf_sampler_data
llama_sampler
llama_sampler_chain_params
llama_sampler_data
llama_sampler_i
user code can implement the interface below in order to create custom llama_sampler
llama_sampler_seq_config
llama_token_data
TODO: simplify (https://github.com/ggml-org/llama.cpp/pull/9294#pullrequestreview-2286561979)
llama_token_data_array
llama_vocab
C interface
LlamaContext
LlamaContextHandle
LlamaCppBindings
Generated bindings for llama.cpp.
LlamaCppDynamicLibraryLoader
LlamaCppNativeLibrary
LlamaModel
LlamaModelHandle
llcs_engine
mtmd_bitmap
mtmd_caps
EXPERIMENTAL API to get mmproj's capabilities without initializing the full context This is only intended to be used by llama-server, breaking changes is expected
mtmd_context
opaque types
mtmd_context_params
mtmd_decoder_pos
mtmd_image_tokens
mtmd_input_chunk
mtmd_input_chunks
mtmd_input_text
UnnamedStruct1
UnnamedStruct2
UnnamedUnion1

Enums

ggml_backend_buffer_usage
Backend buffer
ggml_backend_dev_type
Backend device
ggml_backend_meta_split_axis
ggml_ftype
model file types
ggml_glu_op
ggml_log_level
ggml_numa_strategy
numa strategies
ggml_object_type
ggml_op
available tensor operations:
ggml_op_hint
op hint
ggml_op_pool
ggml_opt_build_type
====== Model / Context ======
ggml_opt_loss_type
built-in loss types, i.e. the built-in quantities minimized by the optimizer custom loss types can be defined via mean or sum which simply reduce the outputs for all datapoints to a single value
ggml_opt_optimizer_type
ggml_prec
precision
ggml_scale_flag
ggml_scale_mode
ggml_sched_priority
scheduling priorities
ggml_sort_order
sort rows
ggml_status
ggml_tensor_flag
this tensor...
ggml_tri_type
ggml_type
NOTE: always add types at the end of the enum to keep backward compatibility
ggml_unary_op
llama_attention_type
llama_context_type
llama_flash_attn_type
llama_ftype
model file types
llama_model_kv_override_type
llama_model_meta_key
llama_pooling_type
llama_rope_scaling_type
llama_rope_type
llama_split_mode
llama_token_attr
llama_token_type
llama_vocab_type
mtmd_input_chunk_type

Typedefs

Dartggml_abort_callback_tFunction = void Function(Pointer<Char> error_message)
Dartggml_abort_callbackFunction = bool Function(Pointer<Void> data)
Dartggml_backend_comm_allreduce_tensor_tFunction = bool Function(Pointer<Void> comm_ctx, Pointer<Pointer<ggml_tensor>> tensors)
Dartggml_backend_comm_free_tFunction = void Function(Pointer<Void> comm_ctx)
Dartggml_backend_comm_init_tFunction = Pointer<Void> Function(Pointer<ggml_backend_t> backends, int n_backends)
Dartggml_backend_eval_callbackFunction = bool Function(int node_index, Pointer<ggml_tensor> t1, Pointer<ggml_tensor> t2, Pointer<Void> user_data)
Dartggml_backend_sched_eval_callbackFunction = bool Function(Pointer<ggml_tensor> t, bool ask, Pointer<Void> user_data)
Dartggml_backend_set_abort_callback_tFunction = void Function(ggml_backend_t backend, ggml_abort_callback abort_callback, Pointer<Void> abort_callback_data)
Dartggml_backend_set_n_threads_tFunction = void Function(ggml_backend_t backend, int n_threads)
Dartggml_backend_split_buffer_type_tFunction = ggml_backend_buffer_type_t Function(int main_device, Pointer<Float> tensor_split)
Dartggml_custom1_op_tFunction = void Function(Pointer<ggml_tensor> dst, Pointer<ggml_tensor> a, int ith, int nth, Pointer<Void> userdata)
Dartggml_custom2_op_tFunction = void Function(Pointer<ggml_tensor> dst, Pointer<ggml_tensor> a, Pointer<ggml_tensor> b, int ith, int nth, Pointer<Void> userdata)
Dartggml_custom3_op_tFunction = void Function(Pointer<ggml_tensor> dst, Pointer<ggml_tensor> a, Pointer<ggml_tensor> b, Pointer<ggml_tensor> c, int ith, int nth, Pointer<Void> userdata)
Dartggml_custom_op_tFunction = void Function(Pointer<ggml_tensor> dst, int ith, int nth, Pointer<Void> userdata)
Dartggml_fp16_t = int
Dartggml_from_float_tFunction = void Function(Pointer<Float> x, Pointer<Void> y, int k)
Dartggml_log_callbackFunction = void Function(ggml_log_level level, Pointer<Char> text, Pointer<Void> user_data)
Dartggml_opt_epoch_callbackFunction = void Function(bool train, ggml_opt_context_t opt_ctx, ggml_opt_dataset_t dataset, ggml_opt_result_t result, int ibatch, int ibatch_max, int t_start_us)
Dartggml_to_float_tFunction = void Function(Pointer<Void> x, Pointer<Float> y, int k)
Dartggml_vec_dot_tFunction = void Function(int n, Pointer<Float> s, int bs, Pointer<Void> x, int bx, Pointer<Void> y, int by, int nrc)
Dartllama_model_set_tensor_data_tFunction = void Function(Pointer<ggml_tensor> tensor, Pointer<Void> userdata)
Dartllama_opt_param_filterFunction = bool Function(Pointer<ggml_tensor> tensor, Pointer<Void> userdata)
Dartllama_pos = int
Dartllama_progress_callbackFunction = bool Function(double progress, Pointer<Void> user_data)
Dartllama_seq_id = int
Dartllama_state_seq_flags = int
Dartllama_token = int
Dartllcs_task_id = int
DynamicLibraryOpener = DynamicLibrary Function(String path)
ggml_abort_callback = Pointer<NativeFunction<ggml_abort_callbackFunction>>
Abort callback If not NULL, called before ggml computation If it returns true, the computation is aborted
ggml_abort_callback_t = Pointer<NativeFunction<ggml_abort_callback_tFunction>>
Function type used in fatal error callbacks
ggml_abort_callback_tFunction = Void Function(Pointer<Char> error_message)
ggml_abort_callbackFunction = Bool Function(Pointer<Void> data)
ggml_backend_buffer_t = Pointer<ggml_backend_buffer>
ggml_backend_buffer_type_t = Pointer<ggml_backend_buffer_type>
ggml_backend_comm_allreduce_tensor_t = Pointer<NativeFunction<ggml_backend_comm_allreduce_tensor_tFunction>>
ggml_backend_comm_allreduce_tensor_tFunction = Bool Function(Pointer<Void> comm_ctx, Pointer<Pointer<ggml_tensor>> tensors)
ggml_backend_comm_free_t = Pointer<NativeFunction<ggml_backend_comm_free_tFunction>>
ggml_backend_comm_free_tFunction = Void Function(Pointer<Void> comm_ctx)
ggml_backend_comm_init_t = Pointer<NativeFunction<ggml_backend_comm_init_tFunction>>
Context management and operations for faster communication between backends, used for tensor parallelism (meta backend)
ggml_backend_comm_init_tFunction = Pointer<Void> Function(Pointer<ggml_backend_t> backends, Size n_backends)
ggml_backend_dev_get_extra_bufts_t = Pointer<NativeFunction<ggml_backend_dev_get_extra_bufts_tFunction>>
Get additional buffer types provided by the device (returns a NULL-terminated array)
ggml_backend_dev_get_extra_bufts_tFunction = Pointer<ggml_backend_buffer_type_t> Function(ggml_backend_dev_t device)
ggml_backend_dev_t = Pointer<ggml_backend_device>
ggml_backend_eval_callback = Pointer<NativeFunction<ggml_backend_eval_callbackFunction>>
ggml_backend_eval_callbackFunction = Bool Function(Int node_index, Pointer<ggml_tensor> t1, Pointer<ggml_tensor> t2, Pointer<Void> user_data)
ggml_backend_event_t = Pointer<ggml_backend_event>
ggml_backend_get_features_t = Pointer<NativeFunction<ggml_backend_get_features_tFunction>>
ggml_backend_get_features_tFunction = Pointer<ggml_backend_feature> Function(ggml_backend_reg_t reg)
ggml_backend_graph_plan_t = Pointer<Void>
ggml_backend_meta_get_split_state_t = Pointer<NativeFunction<ggml_backend_meta_get_split_state_tFunction>>
function to assign split states for statically allocated tensors, compute tensor split states will be assigned to be compatible:
ggml_backend_meta_get_split_state_tFunction = ggml_backend_meta_split_state Function(Pointer<ggml_tensor> tensor, Pointer<Void> userdata)
ggml_backend_reg_t = Pointer<ggml_backend_reg>
ggml_backend_sched_eval_callback = Pointer<NativeFunction<ggml_backend_sched_eval_callbackFunction>>
Evaluation callback for each node in the graph (set with ggml_backend_sched_set_eval_callback) when ask == true, the scheduler wants to know if the user wants to observe this node this allows the scheduler to batch nodes together in order to evaluate them in a single call
ggml_backend_sched_eval_callbackFunction = Bool Function(Pointer<ggml_tensor> t, Bool ask, Pointer<Void> user_data)
ggml_backend_sched_t = Pointer<ggml_backend_sched>
The backend scheduler allows for multiple backend devices to be used together Handles compute buffer allocation, assignment of tensors to backends, and copying of tensors between backends The backends are selected based on:
ggml_backend_set_abort_callback_t = Pointer<NativeFunction<ggml_backend_set_abort_callback_tFunction>>
Set the abort callback for the backend
ggml_backend_set_abort_callback_tFunction = Void Function(ggml_backend_t backend, ggml_abort_callback abort_callback, Pointer<Void> abort_callback_data)
ggml_backend_set_n_threads_t = Pointer<NativeFunction<ggml_backend_set_n_threads_tFunction>>
Set the number of threads for the backend
ggml_backend_set_n_threads_tFunction = Void Function(ggml_backend_t backend, Int n_threads)
ggml_backend_split_buffer_type_t = Pointer<NativeFunction<ggml_backend_split_buffer_type_tFunction>>
Split buffer type for tensor parallelism (old)
ggml_backend_split_buffer_type_tFunction = ggml_backend_buffer_type_t Function(Int main_device, Pointer<Float> tensor_split)
ggml_backend_t = Pointer<ggml_backend>
ggml_custom1_op_t = Pointer<NativeFunction<ggml_custom1_op_tFunction>>
custom operators
ggml_custom1_op_tFunction = Void Function(Pointer<ggml_tensor> dst, Pointer<ggml_tensor> a, Int ith, Int nth, Pointer<Void> userdata)
ggml_custom2_op_t = Pointer<NativeFunction<ggml_custom2_op_tFunction>>
ggml_custom2_op_tFunction = Void Function(Pointer<ggml_tensor> dst, Pointer<ggml_tensor> a, Pointer<ggml_tensor> b, Int ith, Int nth, Pointer<Void> userdata)
ggml_custom3_op_t = Pointer<NativeFunction<ggml_custom3_op_tFunction>>
ggml_custom3_op_tFunction = Void Function(Pointer<ggml_tensor> dst, Pointer<ggml_tensor> a, Pointer<ggml_tensor> b, Pointer<ggml_tensor> c, Int ith, Int nth, Pointer<Void> userdata)
ggml_custom_op_t = Pointer<NativeFunction<ggml_custom_op_tFunction>>
ggml_custom_op_tFunction = Void Function(Pointer<ggml_tensor> dst, Int ith, Int nth, Pointer<Void> userdata)
ggml_fp16_t = Uint16
ieee 754-2008 half-precision float16 todo: make this not an integral type
ggml_from_float_t = Pointer<NativeFunction<ggml_from_float_tFunction>>
ggml_from_float_tFunction = Void Function(Pointer<Float> x, Pointer<Void> y, Int64 k)
ggml_gallocr_t = Pointer<ggml_gallocr>
special tensor flags for use with the graph allocator: ggml_set_input(): all input tensors are allocated at the beginning of the graph in non-overlapping addresses ggml_set_output(): output tensors are never freed and never overwritten
ggml_guid_t = Pointer<Pointer<Uint8>>
ggml_log_callback = Pointer<NativeFunction<ggml_log_callbackFunction>>
TODO these functions were sandwiched in the old optimization interface, is there a better place for them?
ggml_log_callbackFunction = Void Function(UnsignedInt level, Pointer<Char> text, Pointer<Void> user_data)
ggml_opt_context_t = Pointer<ggml_opt_context>
ggml_opt_dataset_t = Pointer<ggml_opt_dataset>
ggml_opt_epoch_callback = Pointer<NativeFunction<ggml_opt_epoch_callbackFunction>>
signature for a callback while evaluating opt_ctx on dataset, called after an evaluation
ggml_opt_epoch_callbackFunction = Void Function(Bool train, ggml_opt_context_t opt_ctx, ggml_opt_dataset_t dataset, ggml_opt_result_t result, Int64 ibatch, Int64 ibatch_max, Int64 t_start_us)
ggml_opt_get_optimizer_params = Pointer<NativeFunction<ggml_opt_get_optimizer_paramsFunction>>
callback to calculate optimizer parameters prior to a backward pass userdata can be used to pass arbitrary data
ggml_opt_get_optimizer_paramsFunction = ggml_opt_optimizer_params Function(Pointer<Void> userdata)
ggml_opt_result_t = Pointer<ggml_opt_result>
ggml_threadpool_t = Pointer<ggml_threadpool>
ggml_to_float_t = Pointer<NativeFunction<ggml_to_float_tFunction>>
ggml_to_float_tFunction = Void Function(Pointer<Void> x, Pointer<Float> y, Int64 k)
ggml_vec_dot_t = Pointer<NativeFunction<ggml_vec_dot_tFunction>>
Internal types and functions exposed for tests and benchmarks
ggml_vec_dot_tFunction = Void Function(Int n, Pointer<Float> s, Size bs, Pointer<Void> x, Size bx, Pointer<Void> y, Size by, Int nrc)
llama_memory_t = Pointer<llama_memory_i>
llama_model_set_tensor_data_t = Pointer<NativeFunction<llama_model_set_tensor_data_tFunction>>
llama_model_set_tensor_data_tFunction = Void Function(Pointer<ggml_tensor> tensor, Pointer<Void> userdata)
llama_opt_param_filter = Pointer<NativeFunction<llama_opt_param_filterFunction>>
function that returns whether or not a given tensor contains trainable parameters
llama_opt_param_filterFunction = Bool Function(Pointer<ggml_tensor> tensor, Pointer<Void> userdata)
llama_pos = Int32
llama_progress_callback = Pointer<NativeFunction<llama_progress_callbackFunction>>
llama_progress_callbackFunction = Bool Function(Float progress, Pointer<Void> user_data)
llama_sampler_context_t = Pointer<Void>
Sampling API
llama_seq_id = Int32
llama_state_seq_flags = Uint32
llama_token = Int32
llcs_task_id = Int64
OperatingSystemProvider = String Function()