generateChatResponseAsync method

  1. @nonVirtual
Stream<ModelResponse> generateChatResponseAsync()

Streams this turn's response token-by-token.

INVARIANT (critical for every consumer): each yielded FunctionCallResponse / ParallelFunctionCallResponse has ALREADY been committed to the persistent chat history (_fullHistory + _modelHistory) at the moment it is yielded — committed before the yield so a tool-response feeds in AFTER it in history order. Because the call is committed on yield, any consumer that collects these calls MUST answer every one (via addQueryChunk with a Message.toolResponse) on EVERY exit path — normal completion, cancellation, a tool throwing, AND a mid-stream stream error — or the committed call is left dangling with no response and poisons the next turn on this reused chat. generateChatResponseWithTools is the reference implementation that balances all four paths; new consumers should prefer it over re-driving this stream directly.

In a plain chat, fails with StateError when the turn's text completes and the session's InferenceModelSession.startTurn reports TurnEnd.closed. Any partial text is committed to fullHistory first. If the text itself fails, the failed-turn take-back rule below applies. An engine whose EngineTurn.end is null cannot expose the close distinction to core; structured chats report it as IncompleteStructuredOutputException as described below. A failure to count the reply's tokens or to rebuild the session after a full context is the stream's error too, after the reply has been committed to fullHistory.

A turn the engine fails commits nothing: the messages staged for it, and any tool call it had committed, are taken back out of fullHistory before the error reaches the caller (for an input too long for the context window the error is a ContextLimitException). A stop is not a failure and keeps what was generated; a turn stopped before any text is taken back as well. Turns an engine runs through ChatTurnHooks.responses keep their own history.

In a chat with a responseFormat the stream yields the reply's text, and a reply that fails the format's check is committed to fullHistory, then the stream's error: a StructuredOutputException. Such a turn always runs core's path; ChatTurnHooks.responses is not called for it. Cancelling the subscription stops the generation.

Core implements this once for every engine (@nonVirtual); an engine changes a chat's turns through ChatTurnHooks.

Implementation

@nonVirtual
Stream<ModelResponse> generateChatResponseAsync() {
  _lastTurnTakenBack = false;
  // Without a session yet, core's stream fails when it is listened to, as
  // it always has.
  final current = _sessionIfAny;
  if (current != null && responseFormatOf(current) != null) {
    return _streamStructuredResponse();
  }
  return (current == null
          ? null
          : _hooks.responses(_engine(current), tokenBuffer: tokenBuffer)) ??
      _streamCoreResponse();
}