estimateTokens method

int estimateTokens(
  1. String text
)

Estimates the token count of text.

A conservative heuristic: four characters per token, rounded up, with a floor of one for non-empty text. Real tokenizers vary between roughly 3 and 5 characters per token depending on language and content, so this errs toward over-counting.

Implementation

int estimateTokens(String text) {
  if (text.isEmpty) return 0;
  return (text.length / 4).ceil();
}