This package is used for flutter_gpt_tokenizer, and should not be used directly.
It implements the BPE algorithm of OpenAI tiktoken using Rust
Libraries
- gpt_tokenizer
- Support for doing something awesome.
gpt_tokenizer package
This package is used for flutter_gpt_tokenizer, and should not be used directly.
It implements the BPE algorithm of OpenAI tiktoken using Rust