hf_tokenizers 0.5.0 
hf_tokenizers: ^0.5.0 copied to clipboard
Published 33 hours ago • 
developeryusuf.com
developeryusuf.comHuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.
11 versions
Stable versions of hf_tokenizers
| Version | Min Dart SDK | Uploaded | Documentation | Archive | |
|---|---|---|---|---|---|
| 0.5.0 | 3.10 | 33 hours ago |  |  | |
| 0.4.5 | 3.10 | 2 days ago | | | |
| 0.4.4 | 3.10 | 2 days ago | | ||
| 0.4.3 | 3.10 | 2 days ago | | ||
| 0.4.2 | 3.10 | 2 days ago | | ||
| 0.4.1 | 3.10 | 2 days ago | | ||
| 0.4.0 | 3.10 | 2 days ago | | ||
| 0.3.0 | 3.10 | 2 days ago | | ||
| 0.2.0 | 3.10 | 3 days ago | | ||
| 0.1.1 | 3.10 | 3 days ago | | ||
| 0.1.0 | 3.10 | 4 days ago | |
Metadata
0
likes
160
points
138
downloads
Documentation
Publisher
Weekly Downloads
Metadata
HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.
Repository (GitHub)
View/report issues
Topics
License
MIT (license)