lib_llama_cpp 0.3.0

App-facing Flutter plugin facade and inference isolate scheduler for llama.cpp.

lib_llama_cpp

App-facing Flutter plugin facade for direct llama.cpp inference.

This package exposes an OpenAI-shaped local client facade backed by a lower-level command stream API. It is the package Flutter applications should depend on directly.

Import

Use the facade API from Flutter app code:

import 'package:lib_llama_cpp/lib_llama_cpp.dart';

Only import the platform interface directly when you need test injection or custom native library resolution:

import 'package:lib_llama_cpp_platform_interface/lib_llama_cpp_platform_interface.dart';

Quick Start

import 'package:lib_llama_cpp/lib_llama_cpp.dart';

final client = LlamaOpenAIClient(
  models: {
    'local': const LlamaModelConfig(modelPath: '/path/to/model.gguf'),
  },
);

final response = await client.responses.create(
  model: 'local',
  input: 'Write one sentence.',
  maxOutputTokens: 16,
);

print(response.outputText);

Streaming uses typed response events with OpenAI-style event names:

await for (final event in client.responses.stream(
  model: 'local',
  input: 'Write one sentence.',
  maxOutputTokens: 16,
)) {
  if (event case LlamaResponseOutputTextDelta(:final delta)) {
    print(delta);
  }
}

Chat-style callers can use the compatibility adapter:

final completion = await client.chat.completions.create(
  model: 'local',
  messages: [
    const LlamaChatMessage(role: 'user', content: 'Write one sentence.'),
  ],
  maxTokens: 16,
);

print(completion.choices.first.message.content);

Advanced Lifecycle API

Use LibLlamaCpp.transform(...) directly when you need command-level lifecycle control or focused tests around model loading, generation, and disposal.

import 'package:lib_llama_cpp/lib_llama_cpp.dart';

final client = const LibLlamaCpp();
final commands = Stream<LlamaCommand>.fromIterable([
  const LlamaLoadModelCommand(modelPath: '/path/to/model.gguf'),
  const LlamaGenerateCommand(prompt: 'Write one sentence.', maxTokens: 16),
  const LlamaDisposeCommand(),
]);

await for (final response in client.transform(commands)) {
  print(response);
}

Public API

The facade exports these public types:

• LlamaOpenAIClient, LlamaModelConfig, LlamaOpenAIException
• LlamaResponseObject, LlamaResponseInputItem, LlamaResponseStreamEvent, LlamaResponseOutputTextDelta, LlamaResponseFailed
• LlamaChatMessage, LlamaChatCompletion
• LlamaEngine, LibLlamaCpp
• LlamaCommand, LlamaLoadModelCommand, LlamaGenerateCommand, LlamaDisposeCommand
• LlamaResponse, LlamaReadyResponse, LlamaStateChangedResponse, LlamaTokenResponse, LlamaErrorResponse, LlamaDoneResponse
• LlamaState

LibLlamaCpp

const LibLlamaCpp({LibLlamaCppPlatform? platform});

Stream<LlamaResponse> transform(
  Stream<LlamaCommand> commands, {
  LlamaState initialState = const LlamaState.empty(),
  LlamaCppLibraryRequest libraryRequest = const LlamaCppLibraryRequest(),
});

transform resolves the platform native library, starts the inference isolate, emits LlamaReadyResponse, then dispatches each command from the input stream. When it receives LlamaDisposeCommand, it emits the disposal responses and stops reading additional commands.

LlamaEngine is the injectable stream interface used by LlamaOpenAIClient. Tests can provide a fake engine that emits LlamaTokenResponse values without a real model.

The optional platform constructor argument and libraryRequest parameter use types from package:lib_llama_cpp_platform_interface. Normal Flutter apps can omit both and rely on federated plugin registration.

Commands

Command	Fields	Current behavior
LlamaLoadModelCommand	modelPath, contextSize, gpuLayerCount	Loads the app-supplied GGUF model path and emits LlamaStateChangedResponse when the runtime state changes.
LlamaGenerateCommand	prompt, maxTokens, temperature, topP, stop	Requires a loaded model. Successful generation emits LlamaTokenResponse values; runtime failures emit LlamaErrorResponse.
LlamaDisposeCommand	none	Resets state to LlamaState.empty(), emits LlamaStateChangedResponse, then emits LlamaDoneResponse.

Responses

Response	Payload
LlamaReadyResponse	library, the resolved LlamaCppLibraryDescriptor
LlamaStateChangedResponse	state, the current LlamaState
LlamaTokenResponse	text and zero-based index for token streaming
LlamaErrorResponse	message
LlamaDoneResponse	none

State

LlamaState tracks the app-facing inference lifecycle:

const LlamaState({String? modelPath, bool isModelLoaded = false});
const LlamaState.empty();

It also provides copyWith({String? modelPath, bool? isModelLoaded}).

Library Resolution

Default Flutter plugin registration chooses the platform resolver. For tests, custom packaging, or local smoke runs, pass a LlamaCppLibraryRequest:

import 'package:lib_llama_cpp/lib_llama_cpp.dart';
import 'package:lib_llama_cpp_platform_interface/lib_llama_cpp_platform_interface.dart';

final responses = const LibLlamaCpp().transform(
  commands,
  libraryRequest: const LlamaCppLibraryRequest(
    preferredPath: '/custom/path/to/libllama.so',
    requiredCapabilities: {LlamaCppLibraryCapability.cpu},
  ),
);

Current platform defaults:

Platform	Resolution	Capabilities
Android	lookup name liblib_llama_cpp_android.so	cpu, vulkan
iOS	path lib_llama_cpp_ios.framework/lib_llama_cpp_ios	cpu, metal
Linux	lookup name liblib_llama_cpp_linux.so	cpu, openBlas, vulkan
macOS	path lib_llama_cpp_macos.framework/lib_llama_cpp_macos	cpu, metal
Windows	lookup name lib_llama_cpp_windows.dll	cpu, vulkan

preferredPath is honored by the current platform resolvers. The requiredCapabilities set is carried in the request API, but the current resolvers return their platform descriptor rather than rejecting unsupported capability requests.

Model Files and Smoke Tests

This package does not download or install models at runtime. Apps and CI runners are responsible for choosing, downloading, verifying, and storing GGUF model files, then passing an app-accessible absolute path through LlamaModelConfig.modelPath or LlamaLoadModelCommand.modelPath.

The example integration smoke is opt-in and runs only when a model path is provided:

cd example
flutter test \
  --dart-define=LIB_LLAMA_CPP_TEST_MODEL=/absolute/path/to/model.gguf \
  integration_test/mobile_smoke_test.dart -d <device-id>

For non-Flutter Dart test entrypoints, LIB_LLAMA_CPP_TEST_MODEL can also be provided through the process environment. CI should keep model download and checksum verification in the runner before invoking package or example tests.