tiktoken_tokenizer_gpt4o_o1 library
Classes
- SpecialTokensSet
- Convenient and type-safe way to represent and manipulate BPE special tokens.
- Tiktoken
- Will encode, decode, and calculate the number of tokens in a text string.
- TiktokenEncoder
- Low level Tiktoken encoder/decoder. It exposes more detailed APIs for processing text using tokens.
- WordCounter
- Counts the number of words of some text. This only works well for languages that use the Latin script.
Enums
- OpenAiModel
- Supported OpenAI models.
- TiktokenEncodingType
- Supported Tiktoken encodings.