generateChatResponseAsync method
Streams this turn's response token-by-token.
INVARIANT (critical for every consumer): each yielded FunctionCallResponse
/ ParallelFunctionCallResponse has ALREADY been committed to the
persistent chat history (_fullHistory + _modelHistory) at the moment it
is yielded — committed before the yield so a tool-response feeds in AFTER
it in history order. Because the call is committed on yield, any consumer
that collects these calls MUST answer every one (via addQueryChunk with a
Message.toolResponse) on EVERY exit path — normal completion,
cancellation, a tool throwing, AND a mid-stream stream error — or the
committed call is left dangling with no response and poisons the next turn
on this reused chat. generateChatResponseWithTools is the reference
implementation that balances all four paths; new consumers should prefer it
over re-driving this stream directly.
In a plain chat, fails with StateError when the turn's text completes and the session's InferenceModelSession.startTurn reports TurnEnd.closed. Any partial text is committed to fullHistory first. If the text itself fails, the failed-turn take-back rule below applies. An engine whose EngineTurn.end is null cannot expose the close distinction to core; structured chats report it as IncompleteStructuredOutputException as described below. A failure to count the reply's tokens or to rebuild the session after a full context is the stream's error too, after the reply has been committed to fullHistory.
A turn the engine fails commits nothing: the messages staged for it, and any tool call it had committed, are taken back out of fullHistory before the error reaches the caller (for an input too long for the context window the error is a ContextLimitException). A stop is not a failure and keeps what was generated; a turn stopped before any text is taken back as well. Turns an engine runs through ChatTurnHooks.responses keep their own history.
In a chat with a responseFormat the stream yields the reply's text, and a reply that fails the format's check is committed to fullHistory, then the stream's error: a StructuredOutputException. Such a turn always runs core's path; ChatTurnHooks.responses is not called for it. Cancelling the subscription stops the generation.
Core implements this once for every engine (@nonVirtual); an engine
changes a chat's turns through ChatTurnHooks.
Implementation
@nonVirtual
Stream<ModelResponse> generateChatResponseAsync() {
_lastTurnTakenBack = false;
// Without a session yet, core's stream fails when it is listened to, as
// it always has.
final current = _sessionIfAny;
if (current != null && responseFormatOf(current) != null) {
return _streamStructuredResponse();
}
return (current == null
? null
: _hooks.responses(_engine(current), tokenBuffer: tokenBuffer)) ??
_streamCoreResponse();
}