hf_tokenizers 0.1.0 copy "hf_tokenizers: ^0.1.0" to clipboard
hf_tokenizers: ^0.1.0 copied to clipboard

Published 4 days ago • verified publisherdeveloperyusuf.com• Latest: 0.5.0
[unknown platforms]
→

Metadata

HuggingFace tokenizers for Dart over FFI. Load any model's tokenizer.json and get byte-exact BPE, byte-level BPE, WordPiece, and Unigram tokenization. Encode text to token ids and decode them back, ba [...]

More...

-/ -
pub points
138
downloads

This package version is not analyzed. Check the latest stable version for its analysis.

Check the analysis log for details.

Weekly downloads

Display as:
By versions:

← Metadata

0
likes
0
points
138
downloads

Publisher

verified publisherdeveloperyusuf.com

Weekly Downloads

Metadata

HuggingFace tokenizers for Dart over FFI. Load any model's tokenizer.json and get byte-exact BPE, byte-level BPE, WordPiece, and Unigram tokenization. Encode text to token ids and decode them back, backed by the Rust tokenizers crate.

Repository (GitHub)
View/report issues

Topics

#llm #ai #tokenizer #ffi #nlp

License

unknown (license)

Dependencies

code_assets, ffi, hooks

More

Packages that depend on hf_tokenizers

Back