v0.3.0 of token_counter

2026-04-25T12:55:23.337661Z

0.3.0 was published on Apr 25, 2026.

Changelog excerpt:
- Added SentencePiece unigram-LM tokenizer in pure Dart.
- New `SpProtoReader`— minimal protobuf decoder for `.model`files (no generated code, no native dependencies).
- New `SpUnigramEncoder`— Viterbi forward-pass segmentation.
- New `SpVocabLoader`abstract class + `BytesSpVocabLoader`implementation.
- New `TokenCounter.loadSpVocab(SpVocabLoader)`— switches the counter to exact SentencePiece mode for `gemini`, `llama`, and `claude`families.
- `TokenCounter.isExact`now returns `true`for both tiktoken BPE and SentencePiece modes.
- 8 new unit tests (proto parsing, Viterbi segmentation, `loadSpVocab`API).