orchestration library
Memory-aware multi-agent orchestration over one loaded model: KV-cache
swapping between agents, GGUF-based memory estimation, dynamic context
sizing, and per-agent disk snapshots (see SPEC.md and
doc/orchestration.md).
Classes
- AgentActivatedEvent
- An agent became the session's resident context.
- AgentHandle
- A registered agent's connection to the orchestrator.
- AgentProfile
- Declares one purpose-specific agent to the orchestrator: identity, persona (instructions), tools, context residency, and sampling.
- AgentRegisteredEvent
- An agent was added to the registry.
- AgentSnapshot
- Metadata about one agent's saved KV-cache snapshot.
- AgentUnregisteredEvent
- An agent was removed from the registry; its KV state (live sequence, stash, disk snapshot) was cleaned up.
- ContextResizedEvent
- The context was recreated at a new size.
- FixedMemoryMonitor
- A monitor that always reports the same snapshot.
- KeepContext
- The current context still fits the budget (or a change is within hysteresis/cooldown); do nothing.
- LlamaOrchestrator
- Owns one loaded model and shares its session between registered agents, sizing the context to the device's memory budget.
- MemoryCritical
- Even the policy's minimum context does not fit the budget.
- MemoryCriticalEvent
- Even the minimum acceptable context no longer fits in memory.
- MemoryPolicy
- Tuning for the orchestrator's memory behavior.
- MemorySnapshot
- A point-in-time view of system memory.
- ModelLoadedEvent
- A model finished loading (or reloading after a resize).
- ModelMemoryEstimate
- What loading a model costs, split into the parts that scale differently.
- OrchestratorEvent
- Telemetry the orchestrator emits so hosts can drive UI ("reloading context…") and logging.
- PlannerDecision
- Outcome of planContextBudget.
- ResizeContext
- Recreate the context at toTokens.
- SnapshotInvalidatedEvent
- A snapshot could not be restored (or no longer fits) and was dropped.
- SnapshotSavedEvent
- An agent's KV state was snapshotted before switching away.
- SnapshotStore
- Web no-op store: wllama cannot restore KV state, so agent switches re-prefill and there is nothing to persist.
- StashEvictedEvent
- A stashed KV state was dropped to enforce the stash byte cap; the agent re-prefills on its next turn.
- StateStashedEvent
- An agent's KV state was copied into the engine's in-memory stash (the fast swap path — no disk involved).
- SystemMemoryMonitor
- Samples system memory on demand.
Enums
- AgentPriority
- How reluctantly an agent's context should be sacrificed under memory pressure. Advisory in v1 (residency is LRU-based); becomes an eviction key when priority-aware eviction lands.
- ResizeReason
- Why a resize was proposed.
Constants
- contextTokenGranularity → const int
- Context sizes are planned in steps of this many tokens, so measurement wobble cannot produce a stream of one-token resize proposals.
- fallbackMemorySnapshot → const MemorySnapshot
- Conservative defaults for platforms with no memory API: assume a small device and let the policy headroom do the rest.
Functions
-
createSystemMemoryMonitor(
) → SystemMemoryMonitor - Web fallback: no memory API worth trusting, so report fixed conservative values (the wasm32 heap cap is the real ceiling there).
-
estimateModelMemory(
{required String architecture, required Map< String, int> metadata, required int modelFileSizeBytes, int draftFileSizeBytes = 0}) → ModelMemoryEstimate? -
Builds a ModelMemoryEstimate from GGUF numeric
metadata(keyed as<architecture>.<suffix>) and artifact file sizes. -
ggufMemoryMetadataKeys(
String architecture) → Set< String> -
GGUF metadata keys the estimator wants for
architecture, suitable forreadGgufMetadata/readGgufMetadataFile. -
planContextBudget(
{required MemorySnapshot memory, required ModelMemoryEstimate estimate, required int currentContextTokens, required int maxContextTokens, required MemoryPolicy policy, int reclaimableBytes = 0, bool underPressure = false, Duration? sinceLastResize}) → PlannerDecision - Plans the context size for the next planning period.