contextTokensFor function

int contextTokensFor(
  1. int maxNumTokens,
  2. BackendKind backend
)

The context maxNumTokens is created with on backend: raised to minCpuGpuContextTokens on the CPU and the GPU, with a log line saying that the context window is not the reply length.

Implementation

int contextTokensFor(int maxNumTokens, BackendKind backend) {
  if (backend == BackendKind.npu || maxNumTokens >= minCpuGpuContextTokens) {
    return maxNumTokens;
  }
  runtimeLog(
    '[LiteRtLmEngine] maxNumTokens ($maxNumTokens) is below the minimum '
    'context size for .litertlm models on the CPU and the GPU; clamping to '
    '$minCpuGpuContextTokens. It is the CONTEXT WINDOW (KV-cache, input + '
    'output), not the reply length: ConversationConfig.maxOutputTokens caps '
    'how many tokens are generated.',
  );
  return minCpuGpuContextTokens;
}