LlamaOrchestrator class

Owns one loaded model and shares its session between registered agents, sizing the context to the device's memory budget.

Agents keep their contexts three ways, best-first:

  1. Multi-sequence (sequenceSlots > 1, native): each resident agent owns a llama.cpp KV sequence inside one context — switching agents costs nothing. With more agents than slots, the least-recently-used resident is stashed (in-memory) and its slot reassigned.
  2. In-memory stash (single slot, native): switching agents copies the outgoing agent's KV state into an engine-side RAM stash and restores the incoming agent's — no disk in the swap path.
  3. Re-prefill (web, or nothing saved): the next generation rebuilds the context from the rendered transcript.

Disk snapshots are written only at persistence points (unload, resize, dispose) for the active agent, and restored when nothing faster exists.

The orchestrator is the owner behind LlamaChatClient's SessionProvider seam: each agent's chat client resolves its session through sessionFor, which activates the agent before returning a session bound to the agent's sequence.

Constructors

LlamaOrchestrator({required LlamaRuntime runtime, required String cacheDirectory, MemoryPolicy policy = const MemoryPolicy(), int sequenceSlots = 1, SystemMemoryMonitor? memoryMonitor, ModelDownloader? downloader, LoggerFactory? loggerFactory})
Creates an orchestrator.

Properties

activeAgentId → String?
The most recently activated agent, or null.
no setter
agentIds → List<String>
The registered agent ids, in registration order.
no setter
agents → List<AgentHandle>
The registered agents, in registration order.
no setter
contextTokens → int
The TOTAL context size the current session was created with; 0 when no model is loaded. Split across sequenceSlots in multi-sequence mode — see contextTokensPerAgent.
no setter
contextTokensPerAgent → int
The context window each agent effectively gets.
no setter
events → Stream<OrchestratorEvent>
Lifecycle and memory telemetry.
no setter
hashCode → int
The hash code for this object.
no setterinherited
isModelLoaded → bool
Whether a model is loaded.
no setter
memoryEstimate → ModelMemoryEstimate?
The memory estimate for the loaded model, when one could be read.
no setter
policy → MemoryPolicy
Memory-behavior tuning.
final
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
sequenceSlots → int
KV sequences the loaded session actually supports (1 until a model is loaded, or on engines without sequence support).
no setter
stashedBytes → int
Bytes currently held by the engine-side KV stash.
no setter

Methods

agent(String id) → AgentHandle?
The handle registered under id, or null.
checkMemory() → Future<void>
Re-plans the context size against a fresh memory sample now, instead of waiting for the next poll tick.
dispose() → Future<void>
Unloads and closes the event stream. The instance is unusable afterwards.
loadModel(ModelSpec spec, {String? localPath, String? localMmprojPath, String? localDraftPath, LlamaLoadProgress? onProgress, ModelMemoryEstimate? estimateOverride, int? sequenceSlots}) → Future<void>
Loads spec, choosing a context size that fits the current memory budget (clamped to spec.contextSize, the policy ceiling, and the model's trained context — all scaled by sequenceSlots).
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
registerAgent(AgentProfile profile) → AgentHandle
Registers profile and returns its handle. Registering an already known id returns the existing handle.
sessionFor(String agentId) → Future<LlamaSession>
Activates agentId and returns a session to generate with (bound to the agent's sequence in multi-sequence mode).
toString() → String
A string representation of this object.
inherited
unloadModel() → Future<void>
Unloads the model (saving the active agent's snapshot to disk first, where supported). Registered agents survive and reactivate after the next loadModel.
unregisterAgent(String id) → Future<bool>
Removes the agent registered under id and cleans up its state: its live sequence (multi-sequence mode), its RAM stash entry, and its disk snapshot. Returns false when no such agent was registered.

Operators

operator ==(Object other) → bool
The equality operator.
inherited