tiktoken 1.0.3

Tiktoken is a BPE tokeniser for use with OpenAI's models. It exposes APIs used for processing text using tokens.

1.0.3

• Refactor: reduce AOT compile size by ~30%

1.0.2

• Add support for GPT-4

1.0.1

• Expose encodingForModel mapping function

1.0.0

• Initial version.