truncateToTokens function

String truncateToTokens(
  1. String content,
  2. int maxTokens
)

Truncate content to roughly maxTokens, keeping the head.

Implementation

String truncateToTokens(String content, int maxTokens) {
  if (roughTokenCountEstimation(content) <= maxTokens) return content;
  final charBudget = maxTokens * 4 - _skillTruncationMarker.length;
  return content.substring(0, min(charBudget, content.length)) +
      _skillTruncationMarker;
}