hf_tokenizers library

HuggingFace tokenizers for Dart, over FFI.

Load any model's tokenizer.json with Tokenizer.fromFile or Tokenizer.fromBytes, then Tokenizer.encode text to token ids and Tokenizer.decode them back. The ids are byte-exact with the reference Rust implementation, so they match what the model was trained on.

Classes

Tokenizer

A tokenizer loaded from a HuggingFace tokenizer.json.

TokenOffset

One token from Tokenizer.encodeWithOffsets: its id and the [start, end) span of the input it came from.