TiktokenEncoder constructor

TiktokenEncoder({
  1. required String name,
  2. required String patternStr,
  3. required Map<ByteArray, int> mergeableRanks,
  4. required Map<String, int> specialTokens,
  5. int? explicitNVocab,
})

Instead of using this constructor, consider using the static helper functions Tiktoken.getEncoder and Tiktoken.getEncoderForModel.

Implementation

TiktokenEncoder({
  required this.name,
  required this.patternStr,
  required this.mergeableRanks,
  required this.specialTokens,
  this.explicitNVocab,
}) {
  maxTokenValue = max(
    mergeableRanks.values.reduce(max),
    specialTokens.values.reduce(max),
  );

  if (explicitNVocab != null) {
    assert(mergeableRanks.length + specialTokens.length == explicitNVocab);
    assert(maxTokenValue == explicitNVocab! - 1);
  }

  specialTokensSet = specialTokens.keys.toSet();

  _coreBPE = CoreBPE.create(mergeableRanks, specialTokens, patternStr);
}