hf_tokenizers 1.2.1 
hf_tokenizers: ^1.2.1 copied to clipboard
Published 5 days ago • 
developeryusuf.com
developeryusuf.comHuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.
22 versions
Stable versions of hf_tokenizers
| Version | Min Dart SDK | Uploaded | Documentation | Archive | |
|---|---|---|---|---|---|
| 1.2.1 | 3.10 | 5 days ago |  |  | |
| 1.2.0 | 3.10 | 6 days ago | | | |
| 1.1.0 | 3.10 | 26 days ago | | ||
| 1.0.6 | 3.10 | 26 days ago | | ||
| 1.0.5 | 3.10 | 26 days ago | | ||
| 1.0.4 | 3.10 | 26 days ago | | ||
| 1.0.3 | 3.10 | 26 days ago | | ||
| 1.0.2 | 3.10 | 27 days ago | | ||
| 1.0.1 | 3.10 | 27 days ago | | ||
| 1.0.0 | 3.10 | 41 days ago | | ||
| 0.5.1 | 3.10 | 43 days ago | | | |
| 0.5.0 | 3.10 | 45 days ago | | ||
| 0.4.5 | 3.10 | 46 days ago | | ||
| 0.4.4 | 3.10 | 46 days ago | | ||
| 0.4.3 | 3.10 | 46 days ago | | ||
| 0.4.2 | 3.10 | 46 days ago | | ||
| 0.4.1 | 3.10 | 46 days ago | | ||
| 0.4.0 | 3.10 | 46 days ago | | ||
| 0.3.0 | 3.10 | 47 days ago | | ||
| 0.2.0 | 3.10 | 47 days ago | | ||
| 0.1.1 | 3.10 | 47 days ago | | ||
| 0.1.0 | 3.10 | 48 days ago | |
Metadata
1
likes
160
points
611
downloads
Documentation
Publisher
Weekly Downloads
Metadata
HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.
Repository (GitHub)
View/report issues
Topics
License
MIT (license)