llm_core library

Core abstractions for LLM (Large Language Model) interactions.

This package provides the foundational interfaces and models used by LLM backend implementations such as Ollama, ChatGPT, and llama.cpp.

Example usage:

import 'package:llm_core/llm_core.dart';

// Use with any LLMChatRepository implementation
void chat(LLMChatRepository repo) async {
  final stream = repo.streamChat('model-name', messages: [
    LLMMessage(role: LLMRole.user, content: 'Hello!')
  ]);
  await for (final chunk in stream) {
    print(chunk.message?.content ?? '');
  }
}

Classes

BackendErrorHandler
Base class for backend-specific error handlers.
CacheKeyGenerator
Utility for generating cache keys from request parameters.
CacheStats
Statistics about cache usage.
ChatRepositoryBuilderBase<T>
Base class for chat repository builders.
DefaultLLMLogger
Default implementation of LLMLogger using the logging package.
DefaultLLMMetrics
Default implementation of LLMMetrics that tracks basic statistics.
ErrorHandlers
Utility functions for error handling in HTTP-based repositories.
HttpClientHelper
Helper class for making HTTP requests with standardized timeout handling and error management.
JsonFormat
Requests JSON output with no schema enforcement.
JsonSchemaFormat
Requests JSON output conforming to a specific JSON Schema.
LLMCapabilities
Capabilities advertised by a repository/model combination.
LLMChatOptions
Options for chat requests.
LLMChatRepository
Abstract repository interface for LLM chat operations.
LLMChunk
Represents a streaming chunk from an LLM response.
LLMChunkMessage
The message portion of an LLM streaming chunk.
LLMEmbedding
Represents a text embedding generated by an LLM.
LLMImageContent
Image content inside an LLMMessage.
LLMInvalidToolCall
A tool call the model emitted whose arguments cannot be decoded.
LLMLogger
Interface for logging in LLM packages.
LLMMessage
Represents a message in an LLM conversation.
LLMMessageContent
A typed content part inside an LLMMessage.
LLMMetrics
Interface for collecting metrics about LLM operations.
LLMResponse
Represents a complete (non-streaming) response from an LLM.
LLMResponseFormat
Structured output format for LLM responses.
LLMTextContent
Text content inside an LLMMessage.
LLMTool
Abstract base class for LLM tools (function calling).
LLMToolCall
Represents a tool call made by an LLM.
LLMToolCallDelta
An incremental fragment of a tool call that is still being streamed.
LLMToolParam
Represents a parameter for an LLM tool.
LLMUsage
Token usage reported by a model provider.
MemoryResponseCache
In-memory response cache implementation.
MergedOptions
Result of merging StreamChatOptions with individual parameters.
RateLimiter
Configuration for rate limiting requests.
RateLimiterUtil
Rate limiter utility for managing request rate limits.
ResponseCache
Interface for caching LLM responses.
RetryConfig
Configuration for retry behavior when making API requests.
RetryUtil
Utility class for retrying operations with exponential backoff.
StreamChatOptions
Backward-compatible name for LLMChatOptions.
StreamChatOptionsMerger
Utility for merging StreamChatOptions with individual parameters.
StreamToolExecutor
Executes tools from LLM chunks and manages the tool execution loop.
TimeoutConfig
Configuration for request timeouts.
TokenBucketRateLimiter
Token bucket rate limiter implementation.
Validation
Validation utilities for LLM requests.
WriteGatedHttpClient
Web stand-in for the dart:io WriteGatedHttpClient.

Enums

LLMFinishReason
Why a model response finished.
LLMLogLevel
Log levels for LLM logging.
LLMRole
The role of a message participant in an LLM conversation.
ReasoningEffort
Portable reasoning-depth levels.

Mixins

LLMRepositoryFeatures
Shared cache and metrics behavior for repository implementations.

Constants

kLLMIdleTimeout → const Duration
Idle timeout applied to pooled connections.
kLLMMaxConcurrentWrites → const int
Default bound on concurrent connect+write phases on kqueue platforms (macOS, iOS).
kLLMMaxConnectionsPerHost → const int
Upper bound on simultaneous connections to one host.

Functions

createLLMHttpClient({TimeoutConfig? timeoutConfig, int maxConnectionsPerHost = kLLMMaxConnectionsPerHost, int? maxConcurrentWrites}) → Client
Creates the HTTP client the LLM backends use by default.
reasoningEffortForBudget(int budget) → ReasoningEffort
Canonical mapping from a reasoning token budget to a portable effort level.

Exceptions / Errors

LLMApiException
Exception thrown when an LLM API request fails.
ModelLoadException
Exception thrown when model loading fails.
ThinkingNotSupportedException
Exception thrown when trying to use thinking on a model that doesn't support it.
ToolLoopIncompleteException
Exception thrown when strict tool-loop mode does not reach a final assistant answer.
ToolsNotSupportedException
Exception thrown when trying to use tools on a model that doesn't support them.
VisionNotSupportedException
Exception thrown when trying to use images/vision on a model that doesn't support it.