v0.2.0 of token_counter

2026-04-25T12:47:58.912002Z

0.2.0 was published on Apr 25, 2026.

Changelog excerpt:
- Added exact tiktoken BPE encoder for `cl100k_base`(GPT-4, GPT-3.5-turbo) and `o200k_base`(GPT-4o, GPT-4.1, o-series) vocabulary families.
- New `TokenCounter.loadVocab(TiktokenVocabLoader)`— supply the raw `.tiktoken`file bytes from any source (Flutter assets, local file, in-memory bytes) to switch the counter into exact mode.
- New `TiktokenVocabLoader`abstract class + `BytesVocabLoader`implementation.
- New `TiktokenVocabParser`for parsing the `.tiktoken`line format.
- New `TiktokenSpecialTokens`constants for `cl100k_base`and `o200k_base`.
- `TokenCounter.isExact`property to distinguish heuristic from BPE mode.
- 13 new unit tests for the BPE algorithm (merge correctness, special tokens, pre-tokenization, `loadVocab`API).