contextTokensFor function
The context maxNumTokens is created with on backend: raised to
minCpuGpuContextTokens on the CPU and the GPU, with a log line saying
that the context window is not the reply length.
Implementation
int contextTokensFor(int maxNumTokens, BackendKind backend) {
if (backend == BackendKind.npu || maxNumTokens >= minCpuGpuContextTokens) {
return maxNumTokens;
}
runtimeLog(
'[LiteRtLmEngine] maxNumTokens ($maxNumTokens) is below the minimum '
'context size for .litertlm models on the CPU and the GPU; clamping to '
'$minCpuGpuContextTokens. It is the CONTEXT WINDOW (KV-cache, input + '
'output), not the reply length: ConversationConfig.maxOutputTokens caps '
'how many tokens are generated.',
);
return minCpuGpuContextTokens;
}