ChatVertexAI class

Wrapper around GCP Vertex AI text chat models API (aka PaLM API for chat).

Example:

final chatModel = ChatVertexAI(
  authHttpClient: authClient,
  project: 'your-project-id',
);
final result = await chatModel([ChatMessage.humanText('Hello')]);

Vertex AI documentation: https://cloud.google.com/vertex-ai/docs/generative-ai/language-model-overview

Set up your Google Cloud Platform project

  1. Select or create a Google Cloud project.
  2. Make sure that billing is enabled for your project.
  3. Enable the Vertex AI API.
  4. Configure the Vertex AI location.

Authentication

To create an instance of ChatVertexAI you need to provide an HTTP client that handles authentication. The easiest way to do this is to use AuthClient from the googleapis_auth package.

There are several ways to obtain an AuthClient depending on your use case. Check out the googleapis_auth package documentation for more details.

Example using a service account JSON:

final serviceAccountCredentials = ServiceAccountCredentials.fromJson(
  json.decode(serviceAccountJson),
);
final authClient = await clientViaServiceAccount(
  serviceAccountCredentials,
  [ChatVertexAI.cloudPlatformScope],
);
final chatVertexAi = ChatVertexAI(
  httpClient: authClient,
  project: 'your-project-id',
);

The service account should have the following permission:

  • aiplatform.endpoints.predict

The requiredOAuth2 scope is:

  • https://www.googleapis.com/auth/cloud-platform (you can use the constant ChatVertexAI.cloudPlatformScope)

See: https://cloud.google.com/vertex-ai/docs/generative-ai/access-control

Available models

  • chat-bison
    • Max input token: 4096
    • Max output tokens: 1024
    • Training data: Up to Feb 2023
    • Max turns: 2500
  • chat-bison-32k
    • Max input and output tokens combined: 32k
    • Training data: Up to Aug 2023
    • Max turns: 2500

The previous list of models may not be exhaustive or up-to-date. Check out the Vertex AI documentation for the latest list of available models.

Model options

You can define default options to use when calling the model (e.g. temperature, stop sequences, etc. ) using the defaultOptions parameter.

The default options can be overridden when calling the model using the options parameter.

Example:

final chatModel = ChatVertexAI(
  httpClient: authClient,
  project: 'your-project-id',
  defaultOptions: ChatVertexAIOptions(
    temperature: 0.9,
  ),
);
final result = await chatModel(
  [ChatMessage.humanText('Hello')],
  options: ChatVertexAIOptions(
    temperature: 0.5,
   ),
);

Constructors

ChatVertexAI.new({required Client httpClient, required String project, String location = 'us-central1', String? rootUrl, ChatVertexAIOptions defaultOptions = const ChatVertexAIOptions(publisher: defaultPublisher, model: defaultModel)})
Wrapper around GCP Vertex AI text chat models API (aka PaLM API for chat).

Properties

client → VertexAIGenAIClient
A client for interacting with Vertex AI API.
final
defaultOptions → ChatVertexAIOptions
The default options to use when invoking the Runnable.
finalinherited
hashCode → int
The hash code for this object.
no setterinherited
modelType → String
Return type of language model.
no setter
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited

Methods

batch(List<PromptValue> inputs, {List<ChatVertexAIOptions>? options}) → Future<List<ChatResult>>
Batches the invocation of the Runnable on the given inputs.
inherited
bind(ChatVertexAIOptions options) → RunnableBinding<PromptValue, ChatVertexAIOptions, ChatResult>
Binds the Runnable to the given options.
inherited
call(List<ChatMessage> messages, {ChatVertexAIOptions? options}) → Future<AIChatMessage>
Runs the chat model on the given messages and returns a chat message.
inherited
close() → void
Cleans up any resources associated with it the Runnable.
inherited
countTokens(PromptValue promptValue, {ChatVertexAIOptions? options}) → Future<int>
Returns the number of tokens resulting from tokenize the given prompt.
getCompatibleOptions(RunnableOptions? options) → ChatVertexAIOptions?
Returns the given options if they are compatible with the Runnable, otherwise returns null.
inherited
invoke(PromptValue input, {ChatVertexAIOptions? options}) → Future<ChatResult>
Invokes the Runnable on the given input.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
pipe<NewRunOutput extends Object?, NewCallOptions extends RunnableOptions>(Runnable<ChatResult, NewCallOptions, NewRunOutput> next) → RunnableSequence<PromptValue, NewRunOutput>
Pipes the output of this Runnable into another Runnable using a RunnableSequence.
inherited
stream(PromptValue input, {ChatVertexAIOptions? options}) → Stream<ChatResult>
Streams the output of invoking the Runnable on the given input.
inherited
streamFromInputStream(Stream<PromptValue> inputStream, {ChatVertexAIOptions? options}) → Stream<ChatResult>
Streams the output of invoking the Runnable on the given inputStream.
inherited
tokenize(PromptValue promptValue, {ChatVertexAIOptions? options}) → Future<List<int>>
Tokenizes the given prompt using the encoding used by the language model.
toString() → String
A string representation of this object.
inherited
withFallbacks(List<Runnable<PromptValue, RunnableOptions, ChatResult>> fallbacks) → RunnableWithFallback<PromptValue, ChatResult>
Adds fallback runnables to be invoked if the primary runnable fails.
inherited
withRetry({int maxRetries = 3, FutureOr<bool> retryIf(Object e)?, List<Duration?>? delayDurations, bool addJitter = false}) → RunnableRetry<PromptValue, ChatResult>
Adds retry logic to an existing runnable.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited

Constants

cloudPlatformScope → const String
Scope required for Vertex AI API calls.
defaultModel → const String
The default model to use unless another is specified.
defaultPublisher → const String
The default publisher to use unless another is specified.