tiktoken/src/core_bpe library

Classes

CoreBPE

BPE tokeniser

Functions

decodeLastUtf8(List<int> slice) → _Tuple2<String?, int>

UTF-8 decode a single Unicode scalar value from the end of a slice.

isWhitespace(String? c) → bool