hf_tokenizers 0.2.0 copy "hf_tokenizers: ^0.2.0" to clipboard
hf_tokenizers: ^0.2.0 copied to clipboard

Published 3 days ago • verified publisherdeveloperyusuf.com• Latest: 0.5.0
[unknown platforms]
→

Metadata

HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.

More...

-/ -
pub points
138
downloads

This package version is not analyzed. Check the latest stable version for its analysis.

Check the analysis log for details.

Weekly downloads

Display as:
By versions:

← Metadata

0
likes
0
points
138
downloads

Publisher

verified publisherdeveloperyusuf.com

Weekly Downloads

Metadata

HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.

Repository (GitHub)
View/report issues

Topics

#llm #ai #tokenizer #ffi #nlp

License

unknown (license)

Dependencies

code_assets, ffi, hooks

More

Packages that depend on hf_tokenizers

Back