tiktoken/src/tiktoken_encoder library

Classes

TiktokenEncoder

Low level Tiktoken encoder/decoder. It exposes more detailed APIs for processing text using tokens.