ChatOllama class

Wrapper around Ollama Chat API that enables to interact with the LLMs in a chat-like fashion.

Ollama allows you to run open-source large language models, such as Llama 3.2, Gemma 2 or LLaVA, locally.

For a complete list of supported models and model variants, see the Ollama model library.

Example:

final chatModel = ChatOllama();
final messages = [
  ChatMessage.system('You are a helpful assistant that translates English to French.'),
  ChatMessage.humanText('I love programming.'),
];
final prompt = PromptValue.chat(messages);
final res = await llm.invoke(prompt);

Ollama API docs

Setup

Download and install Ollama
Fetch a model via ollama pull <model family>

e.g., for Llama 3: ollama pull llama3.2

Ollama base URL

By default, ChatOllama uses 'http://localhost:11434/api' as base URL (default Ollama API URL). But if you are running Ollama on a different one, you can override it using the baseUrl parameter.

Call options

You can configure the parameters that will be used when calling the chat completions API in several ways:

Default options:

Use the defaultOptions parameter to set the default options. These options will be used unless you override them when generating completions.

final chatModel = ChatOllama(
  defaultOptions: const ChatOllamaOptions(
    model: 'llama3.2',
    temperature: 0,
    format: 'json',
  ),
);

Call options:

You can override the default options when invoking the model:

final res = await chatModel.invoke(
  prompt,
  options: const ChatOllamaOptions(seed: 9999),
);

Bind:

You can also change the options in a Runnable pipeline using the bind method.

In this example, we are using two totally different models for each question:

final chatModel = ChatOllama();
const outputParser = StringOutputParser();
final prompt1 = PromptTemplate.fromTemplate('How are you {name}?');
final prompt2 = PromptTemplate.fromTemplate('How old are you {name}?');
final chain = Runnable.fromMap({
  'q1': prompt1 | chatModel.bind(const ChatOllamaOptions(model: 'llama3.2')) | outputParser,
  'q2': prompt2| chatModel.bind(const ChatOllamaOptions(model: 'mistral')) | outputParser,
});
final res = await chain.invoke({'name': 'David'});

Advance

Custom HTTP client

You can always provide your own implementation of http.Client for further customization:

final client = ChatOllama(
  client: MyHttpClient(),
);

Using a proxy

HTTP proxy

You can use your own HTTP proxy by overriding the baseUrl and providing your required headers:

final client = ChatOllama(
  baseUrl: 'https://my-proxy.com',
  headers: {'x-my-proxy-header': 'value'},
  queryParams: {'x-my-proxy-query-param': 'value'},
);

If you need further customization, you can always provide your own http.Client.

SOCKS5 proxy

To use a SOCKS5 proxy, you can use the socks5_proxy package and a custom http.Client.

Constructors

ChatOllama({String baseUrl = 'http://localhost:11434/api', Map<String, String>? headers, Map<String, dynamic>? queryParams, Client? client, ChatOllamaOptions defaultOptions = const ChatOllamaOptions(model: defaultModel), String encoding = 'cl100k_base'}): Create a new ChatOllama instance.

Properties

defaultOptions → ChatOllamaOptions: The default options to use when invoking the Runnable.
finalinherited
encoding ↔ String: The encoding to use by tiktoken when tokenize is called.
getter/setter pair
hashCode → int: The hash code for this object.
no setterinherited
modelType → String: Return type of language model.
no setter
runtimeType → Type: A representation of the runtime type of the object.
no setterinherited

Methods

batch(List<PromptValue> inputs, {List<ChatOllamaOptions>? options}) → Future<List<ChatResult>>: Batches the invocation of the Runnable on the given inputs.
inherited
bind(ChatOllamaOptions options) → RunnableBinding<PromptValue, ChatOllamaOptions, ChatResult>: Binds the Runnable to the given options.
inherited
call(List<ChatMessage> messages, {ChatOllamaOptions? options}) → Future<AIChatMessage>: Runs the chat model on the given messages and returns a chat message.
inherited
close() → void: Cleans up any resources associated with it the Runnable.
countTokens(PromptValue promptValue, {ChatOllamaOptions? options}) → Future<int>: Returns the number of tokens resulting from tokenize the given prompt.
inherited
getCompatibleOptions(RunnableOptions? options) → ChatOllamaOptions?: Returns the given options if they are compatible with the Runnable, otherwise returns null.
inherited
invoke(PromptValue input, {ChatOllamaOptions? options}) → Future<ChatResult>: Invokes the Runnable on the given input.
noSuchMethod(Invocation invocation) → dynamic: Invoked when a nonexistent method or property is accessed.
inherited
pipe<NewRunOutput extends Object?, NewCallOptions extends RunnableOptions>(Runnable<ChatResult, NewCallOptions, NewRunOutput> next) → RunnableSequence<PromptValue, NewRunOutput>: Pipes the output of this Runnable into another Runnable using a RunnableSequence.
inherited
stream(PromptValue input, {ChatOllamaOptions? options}) → Stream<ChatResult>: Streams the output of invoking the Runnable on the given input.
streamFromInputStream(Stream<PromptValue> inputStream, {ChatOllamaOptions? options}) → Stream<ChatResult>: Streams the output of invoking the Runnable on the given inputStream.
inherited
tokenize(PromptValue promptValue, {ChatOllamaOptions? options}) → Future<List<int>>: Tokenizes the given prompt using tiktoken.
toString() → String: A string representation of this object.
inherited
withFallbacks(List<Runnable<PromptValue, RunnableOptions, ChatResult>> fallbacks) → RunnableWithFallback<PromptValue, ChatResult>: Adds fallback runnables to be invoked if the primary runnable fails.
inherited
withRetry({int maxRetries = 3, FutureOr<bool> retryIf(Object e)?, List<Duration?>? delayDurations, bool addJitter = false}) → RunnableRetry<PromptValue, ChatResult>: Adds retry logic to an existing runnable.
inherited

Operators

operator ==(Object other) → bool: The equality operator.
inherited

Constants

defaultModel → const String: The default model to use unless another is specified.