truncateToTokens method

String truncateToTokens(String text, int maxTokens)

Truncates text to at most maxTokens, returning the truncated string.

Binary-searches for the character boundary that yields maxTokens.

Implementation

String truncateToTokens(String text, int maxTokens) {
  if (_encoder.count(text) <= maxTokens) return text;
  // Binary search for the right length.
  var lo = 0;
  var hi = text.length;
  while (lo < hi) {
    final mid = (lo + hi + 1) ~/ 2;
    if (_encoder.count(text.substring(0, mid)) <= maxTokens) {
      lo = mid;
    } else {
      hi = mid - 1;
    }
  }
  return text.substring(0, lo);
}