langchain_tiktoken library
tiktoken is a BPE tokeniser for use with OpenAI's models. It exposes APIs used to process text using tokens.
Classes
- SpecialTokensSet
- Convenient and type-safe way to represent and manipulate BPE special tokens.
- Tiktoken
- Tiktoken encoder/decoder.
Functions
-
encodingForModel(
String modelName) → Tiktoken - Returns the encoding used by a model
-
getEncoding(
String encodingName) → Tiktoken -
listEncodingNames(
) → Set< String> - Returns all avalilable encoding names