encodeSingleToken method

int encodeSingleToken(List<int> bytes)

Encodes text corresponding to a single token to its token value.

NOTE: this will encode all special tokens.

Throws TiktokenError if the token is not in the vocabulary.

Example:

final enc = getEncoding("gpt2");
enc.encodeSingleToken("hello") // 31373

Implementation

int encodeSingleToken(List<int> bytes) {
  return _coreBPE.encodeSingleToken(ByteArray.fromList(bytes));
}