applyThinkingTokenBudget function
- @visibleForTesting
- LiteRtLmBindings b,
- Pointer<
LiteRtLmConversationOptionalArgs> optionalArgs, { - required bool thinking,
- required int? budget,
Gives one send's optionalArgs a thinking token budget of budget when
the send thinkings and a budget is set; otherwise touches nothing, so a
send without a budget reaches LiteRT-LM exactly as before.
LiteRT-LM applies the budget at decode time to the conversation's first
channel (thought on the Gemma 4 bundles); a bundle without that channel
ignores it. The args keep a copy of the config, which is deleted here.
Implementation
@visibleForTesting
void applyThinkingTokenBudget(
LiteRtLmBindings b,
Pointer<LiteRtLmConversationOptionalArgs> optionalArgs, {
required bool thinking,
required int? budget,
}) {
if (!thinking || budget == null) return;
// A failure here throws before any generation starts.
final config = createLiteRtLm(
'litert_lm_thinking_config_create',
b.litert_lm_thinking_config_create,
b.lastError,
);
try {
checkLiteRtLmSetter(
'litert_lm_thinking_config_set_enable_thinking',
b.litert_lm_thinking_config_set_enable_thinking(config, true),
b.lastError,
);
checkLiteRtLmSetter(
'litert_lm_thinking_config_set_thinking_token_budget',
b.litert_lm_thinking_config_set_thinking_token_budget(config, budget),
b.lastError,
);
checkLiteRtLmSetter(
'litert_lm_conversation_optional_args_set_thinking_config',
b.litert_lm_conversation_optional_args_set_thinking_config(
optionalArgs,
config,
),
b.lastError,
);
} finally {
b.litert_lm_thinking_config_delete(config);
}
}