generateChatResponseWithTools method
Drive flutter_gemma's function-calling loop to completion. Stream this
turn's text/thinking tokens; whenever the model calls a tool, run
onToolCall and feed its result back as a tool-response message, then
continue — until a turn has no calls (the model's final answer) or
maxToolTurns / isCancelled stops it.
PRECONDITION: the user message must already be staged (call addQueryChunk
first), same as generateChatResponseAsync. onToolCall returns the
{...} response map fed back via Message.toolResponse. Tool execution
is the caller's (tools are app actions); core only parses the call and
drives the loop. onMaxToolTurns, when supplied, is invoked once if the
loop exhausts maxToolTurns without a call-free answer — so a consumer
driving this loop can surface its own terminal signal (e.g. AgentLoop's
MaxIterationsEvent).
Implementation
Stream<ModelResponse> generateChatResponseWithTools({
required FutureOr<Map<String, dynamic>> Function(FunctionCallResponse call)
onToolCall,
int maxToolTurns = 8,
bool Function()? isCancelled,
void Function()? onMaxToolTurns,
}) async* {
if (maxToolTurns < 1) {
// A cap below 1 would exit before any generation — the already-staged
// user turn would get NO response and the stream would close empty (a
// silent no-op). Fail loud (asserts are stripped in release).
throw RangeError.range(maxToolTurns, 1, null, 'maxToolTurns');
}
for (var turn = 0; turn < maxToolTurns; turn++) {
if (isCancelled?.call() ?? false) return;
final pending = <FunctionCallResponse>[];
try {
await for (final r in generateChatResponseAsync()) {
// Exhaustive over the sealed ModelResponse: a future subtype fails to
// compile here instead of silently passing through as text.
switch (r) {
case FunctionCallResponse():
pending.add(r);
case ParallelFunctionCallResponse(:final calls):
pending.addAll(calls);
case TextResponse() || ThinkingResponse():
yield r; // pass through to the caller's text/thinking stream
}
}
} catch (_) {
// generateChatResponseAsync commits each tool-call to the persistent
// history the moment it yields it (see the callsites above). A mid-stream
// decode error rethrows past the balancing below, leaving a committed
// call with no tool-response — it dangles and poisons the next turn on
// this reused chat. Answer the collected calls first, then rethrow the
// ORIGINAL error (the failure still surfaces to the caller — never
// hidden).
if (pending.isNotEmpty) {
gemmaLog(
'InferenceChat.generateChatResponseWithTools: generation stream '
'errored mid-turn; balancing ${pending.length} committed '
'tool-call(s) before rethrowing.',
);
}
try {
await _answerToolCalls(pending, const {
'status': 'failed',
'error': 'generation stream errored before this tool call ran',
});
} catch (balanceError) {
// The already-errored session rejected the balancing feed too. Do NOT
// let that mask the original failure — log loudly and still rethrow the
// original below. The caller's safe recovery is to recreate the session
// / clearHistory(replayHistory:), which the Dart-side history makes
// correct once the session is usable again.
gemmaLog(
'InferenceChat.generateChatResponseWithTools: could not balance '
'committed tool-call(s) after a stream error ($balanceError); the '
'persistent chat history may be left unbalanced — recover by '
'recreating the session.',
);
}
rethrow;
}
if (pending.isEmpty) return; // model's final (call-free) answer
// generateChatResponseAsync already committed the assistant tool-call(s)
// to history; EVERY committed call must get a matching tool-response or
// the (persistent, reused-across-turns) chat is left with a dangling
// call that poisons the next turn (the model re-issues it or replies
// empty). So both abnormal exits below balance the history first.
if (isCancelled?.call() ?? false) {
// Barge-in after generation, before execution: don't run the tools,
// but still answer the committed call(s) with a cancelled marker.
await _answerToolCalls(pending, const {'status': 'cancelled'});
return;
}
for (var i = 0; i < pending.length; i++) {
if (isCancelled?.call() ?? false) {
// Barge-in landed mid-turn (between individual tool calls). Don't run
// the remaining tools' side effects — mirrors AgentLoop's between-call
// check (agent_loop.dart) — but answer every not-yet-run call (i..end)
// with a cancelled marker so no committed call is left dangling, then
// stop. On the voice detach path the per-turn token stays cancelled
// forever, so a still-running DETACHED loop bails here at the next
// tool boundary instead of executing the rest of this turn's tools.
await _answerToolCalls(pending.sublist(i), const {
'status': 'cancelled',
});
return;
}
final call = pending[i];
final Map<String, dynamic> response;
try {
response = await onToolCall(call);
} catch (e) {
// The app's tool threw. Answer the failed call (and any siblings in
// this turn not yet run) so the committed call isn't left dangling,
// then rethrow so the caller sees the failure.
await addQueryChunk(
Message.toolResponse(
toolName: call.name,
response: {'error': '$e'},
),
);
await _answerToolCalls(pending.sublist(i + 1), const {
'status': 'not run — an earlier tool call in this turn failed',
});
rethrow;
}
await addQueryChunk(
Message.toolResponse(toolName: call.name, response: response),
);
}
}
// Exhausted maxToolTurns with calls still pending: each turn's calls WERE
// answered, but no final call-free generation ran, so the reply may be
// empty/truncated. Log it (and notify [onMaxToolTurns], so a consumer that
// drives this loop can surface its own terminal — e.g. AgentLoop's
// MaxIterationsEvent) — silent truncation would violate the
// no-masking-failure rule.
onMaxToolTurns?.call();
gemmaLog(
'InferenceChat.generateChatResponseWithTools: hit maxToolTurns '
'($maxToolTurns) with tool calls still pending; stopping. The reply may '
'be empty or truncated — the model never produced a call-free answer.',
);
}