langchain_tiktoken library

tiktoken is a BPE tokeniser for use with OpenAI's models. It exposes APIs used to process text using tokens.

Classes

SpecialTokensSet
Convenient and type-safe way to represent and manipulate BPE special tokens.
Tiktoken
Tiktoken encoder/decoder.

Functions

encodingForModel(String modelName) Tiktoken
Returns the encoding used by a model
getEncoding(String encodingName) Tiktoken
listEncodingNames() Set<String>
Returns all avalilable encoding names