hf_tokenizers 1.1.0 
hf_tokenizers: ^1.1.0 copied to clipboard
Published 3 days ago • 
developeryusuf.com
developeryusuf.comHuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.
20 versions
Stable versions of hf_tokenizers
| Version | Min Dart SDK | Uploaded | Documentation | Archive | |
|---|---|---|---|---|---|
| 1.1.0 | 3.10 | 3 days ago |  |  | |
| 1.0.6 | 3.10 | 3 days ago | | | |
| 1.0.5 | 3.10 | 3 days ago | | ||
| 1.0.4 | 3.10 | 3 days ago | | ||
| 1.0.3 | 3.10 | 3 days ago | | ||
| 1.0.2 | 3.10 | 4 days ago | | ||
| 1.0.1 | 3.10 | 4 days ago | | ||
| 1.0.0 | 3.10 | 18 days ago | | ||
| 0.5.1 | 3.10 | 20 days ago | | | |
| 0.5.0 | 3.10 | 22 days ago | | ||
| 0.4.5 | 3.10 | 22 days ago | | ||
| 0.4.4 | 3.10 | 23 days ago | | ||
| 0.4.3 | 3.10 | 23 days ago | | ||
| 0.4.2 | 3.10 | 23 days ago | | ||
| 0.4.1 | 3.10 | 23 days ago | | ||
| 0.4.0 | 3.10 | 23 days ago | | ||
| 0.3.0 | 3.10 | 23 days ago | | ||
| 0.2.0 | 3.10 | 24 days ago | | ||
| 0.1.1 | 3.10 | 24 days ago | | ||
| 0.1.0 | 3.10 | 24 days ago | |
Metadata
0
likes
160
points
1.06k
downloads
Documentation
Publisher
Weekly Downloads
Metadata
HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.
Repository (GitHub)
View/report issues
Topics
License
MIT (license)