dart_bert_tokenizer 1.1.0

A pure Dart BERT WordPiece tokenizer with Hugging Face tokenizer.json loading, Unicode offsets, padding, truncation, and verified model fixtures.

dart_bert_tokenizer

A lightweight, pure Dart implementation of BERT WordPiece tokenizer.

Features

• Pure Dart - Zero runtime dependencies for Flutter native, server and CLI applications
• Memory Efficient - Typed arrays (Int32List, Uint8List) for token IDs and masks
• Full API - Encoding, decoding, padding, truncation, offset mapping
• Batch Processing - Sequential and parallel (Isolate-based) batch encoding
• HuggingFace tokenizer.json - Load directly from HuggingFace tokenizer files
• Well Tested - Offline HF goldens and pinned network fixtures for eleven supported model pipelines and two unsupported-pipeline boundaries

Installation

dependencies:
  dart_bert_tokenizer: ^1.1.0

Quick Start

The token strings and IDs below assume the google-bert/bert-base-uncased vocabulary. Other models have different vocabularies and special-token IDs.

import 'package:dart_bert_tokenizer/dart_bert_tokenizer.dart';

void main() {
  // Load tokenizer
  final tokenizer = WordPieceTokenizer.fromVocabFileSync('vocab.txt');

  // Encode text
  final encoding = tokenizer.encode('Hello, world!');
  print(encoding.tokens); // [[CLS], hello, ,, world, !, [SEP]]
  print(encoding.ids);    // [101, 7592, 1010, 2088, 999, 102]

  // Decode back to text
  final text = tokenizer.decode(encoding.ids, skipSpecialTokens: true);
  print(text); // hello , world !
}

Loading from tokenizer.json

Load supported Hugging Face WordPiece tokenizer.json files. Normalization, post-processing, decoder, padding and truncation settings are extracted; unsupported configurations raise FormatException. See compatibility and limits.

// From file (async)
final tokenizer = await WordPieceTokenizer.fromTokenizerJson('tokenizer.json');

// From file (sync)
final tokenizer = WordPieceTokenizer.fromTokenizerJsonSync('tokenizer.json');

// From JSON string (e.g., embedded asset)
final tokenizer = WordPieceTokenizer.fromTokenizerJsonString(jsonString);

// Override extracted config if needed
final tokenizer = WordPieceTokenizer.fromTokenizerJsonSync(
  'tokenizer.json',
  configOverride: WordPieceConfig(lowercase: false),
);

configOverride replaces the exposed WordPiece settings as a whole, including defaults for fields you omit; it is not a partial merge. It also replaces the JSON template with the legacy CLS/SEP configuration. Prefer loading without an override when matching HF output.

Usage

Single Text Encoding

final encoding = tokenizer.encode('Hello world');

print(encoding.tokens);        // Token strings
print(encoding.ids);           // Token IDs (Int32List)
print(encoding.attentionMask); // Attention mask (Uint8List)
print(encoding.typeIds);       // Type IDs (Uint8List)
print(encoding.offsets);       // Original code-point offsets [(start, end), ...]
print(encoding.wordIds);       // Word indices
print(encoding.sequenceIds);   // Sequence indices (0, 1, or null)

// Without special tokens
final raw = tokenizer.encode('Hello', addSpecialTokens: false);

Sentence Pair Encoding

// For QA, NLI, sentence similarity tasks
final encoding = tokenizer.encodePair(
  'What is machine learning?',
  'Machine learning is a subset of AI.',
);

print(encoding.typeIds);     // Token type IDs assigned by the template
print(encoding.sequenceIds); // 0/1 for input A/B; null for special/padding tokens

Batch Encoding

// Sequential batch
final encodings = tokenizer.encodeBatch(['Hello', 'World', 'Test']);

// Parallel batch (uses Isolates for batches >= 8)
final parallel = await tokenizer.encodeBatchParallel(texts);

// Pair batch
final pairs = [('Q1', 'A1'), ('Q2', 'A2')];
final pairEncodings = tokenizer.encodePairBatch(pairs);
final parallelPairs = await tokenizer.encodePairBatchParallel(pairs);

Padding

// Fluent API
final tokenizer = WordPieceTokenizer.fromVocabFileSync('vocab.txt')
  ..enablePadding(length: 512, direction: PaddingDirection.right);

// Or pad to longest in batch
tokenizer.enablePadding(); // Auto-pads to longest

// Manual padding
final padded = encoding.withPadding(
  targetLength: 128,
  padTokenId: tokenizer.vocab.padTokenId,
  padOnRight: true,
);

// Pad to multiple of N
final paddedToMultiple = encoding.withPaddingToMultipleOf(
  multiple: 8,
  padTokenId: tokenizer.vocab.padTokenId,
);

Truncation

// Fluent API
final tokenizer = WordPieceTokenizer.fromVocabFileSync('vocab.txt')
  ..enableTruncation(maxLength: 512, direction: TruncationDirection.right);

// Manual truncation
final truncated = encoding.withTruncation(maxLength: 64);

// Truncation strategies for pairs
tokenizer.encodePair(textA, textB,
  maxLength: 128,
  truncationStrategy: TruncationStrategy.longestFirst,
);

Truncation Strategies:

• longestFirst - Remove from longest sequence iteratively
• onlyFirst - Truncate first sequence only
• onlySecond - Truncate second sequence only
• doNotTruncate - No truncation

The tokenizer truncates content before inserting special tokens; maxLength includes those special tokens. onlyFirst/onlySecond throw when the selected sequence cannot remove enough tokens. Encoding.withTruncation() instead slices an already-built encoding and can remove special tokens.

Offset Mapping

final encoding = tokenizer.encode('Hello world');

// Original Unicode code-point position -> Token index
final tokenIdx = encoding.charToToken(6); // 'w' -> token index

// Token index -> Character span
final (start, end) = encoding.tokenToChars(1)!; // token -> (0, 5)

// Word index -> Token span
final (startToken, endToken) = encoding.wordToTokens(0)!;

// Token -> Word index
final wordIdx = encoding.tokenToWord(1);

// Token -> Sequence index (0, 1, or null for special tokens)
final seqIdx = encoding.tokenToSequence(1);

Vocabulary Access

print(tokenizer.vocab.size);       // 30522
print(tokenizer.vocab.clsTokenId); // 101
print(tokenizer.vocab.sepTokenId); // 102
print(tokenizer.vocab.padTokenId); // 0
print(tokenizer.vocab.unkTokenId); // 100
print(tokenizer.vocab.maskTokenId); // 103

// Token <-> ID conversion
tokenizer.convertTokensToIds(['hello', 'world']); // [7592, 2088]
tokenizer.convertIdsToTokens([7592, 2088]);       // ['hello', 'world']

// Check if token exists
tokenizer.vocab.contains('hello'); // true

Decoding

// Decode with special tokens
final text = tokenizer.decode(encoding.ids, skipSpecialTokens: false);

// Decode without special tokens (default: true)
final withoutSpecials = tokenizer.decode(encoding.ids);

// Batch decode
final texts = tokenizer.decodeBatch(idsBatch);

ONNX Runtime Integration

Use with ONNX Runtime for on-device ML inference:

import 'package:dart_bert_tokenizer/dart_bert_tokenizer.dart';
import 'dart:typed_data';

final tokenizer = WordPieceTokenizer.fromVocabFileSync('vocab.txt')
  ..enableTruncation(maxLength: 512);

final encoding = tokenizer.encode('search_query: What is ML?');

// Encoding.ids is already Int32List, convert to Int64List for ONNX
final inputIds = Int64List.fromList(encoding.ids);
final attentionMask = Int64List.fromList(encoding.attentionMask);

// Pass to ONNX session
// final outputs = await session.run({
//   'input_ids': inputIds,
//   'attention_mask': attentionMask,
// });

Configuration

final tokenizer = WordPieceTokenizer(
  vocab: vocab,
  config: WordPieceConfig(
    lowercase: true,           // Convert to lowercase (default: true)
    stripAccents: true,        // Remove accents (default: true)
    handleChineseChars: true,  // Space around Chinese chars (default: true)
    subwordPrefix: '##',       // Subword prefix (default: '##')
    maxWordLength: 200,        // Max word length before [UNK] (default: 200)
    addClsToken: true,         // Add [CLS] token (default: true)
    addSepToken: true,         // Add [SEP] token (default: true)
  ),
);

API Reference

WordPieceTokenizer

Method	Description
fromVocabFile(path)	Load from vocab file (async)
fromVocabFileSync(path)	Load from vocab file (sync)
fromTokenizerJson(path)	Load from tokenizer.json (async)
fromTokenizerJsonSync(path)	Load from tokenizer.json (sync)
fromTokenizerJsonString(json)	Load from JSON string
encode(text)	Encode single text
encodePair(textA, textB)	Encode text pair
encodeBatch(texts)	Encode multiple texts
encodeBatchParallel(texts)	Parallel batch encoding
encodePairBatch(pairs)	Batch encoding of text pairs
encodePairBatchParallel(pairs)	Parallel encoding of text pairs
decode(ids)	Decode IDs to text
decodeBatch(idsBatch)	Batch decode
enablePadding() / noPadding()	Configure padding
enableTruncation() / noTruncation()	Configure truncation
convertTokensToIds(tokens)	Convert tokens to IDs
convertIdsToTokens(ids)	Convert IDs to tokens
numSpecialTokensToAdd(isPair: true)	Get special token count

Encoding

Property	Type	Description
tokens	List<String>	Token strings
ids	Int32List	Token IDs
attentionMask	Uint8List	Attention mask (1=attend, 0=ignore)
typeIds	Uint8List	Template-defined token type IDs (0–255)
specialTokensMask	Uint8List	Special token mask
offsets	List<(int, int)>	Original Unicode code-point offsets
wordIds	List<int?>	Word indices local to each input sequence
sequenceIds	List<int?>	Sequence indices
length	int	Number of tokens

Performance

Run the benchmarks below on your target device and vocabulary. This release does not publish hardware-independent throughput or memory guarantees.

Vocabulary Files

You can load from either vocab.txt or tokenizer.json:

Format	Method	Description
vocab.txt	fromVocabFile() / fromVocabFileSync()	One token per line, zero-based line index = ID
tokenizer.json	fromTokenizerJson() / fromTokenizerJsonSync()	Supported Hugging Face WordPiece pipeline config

Download from HuggingFace:

• bert-base-uncased vocab.txt
• bert-base-uncased tokenizer.json

Testing

# Run offline tests (network fixtures are opt-in)
dart test

# Run specific test file
dart test test/hf_compatible_test.dart

# Run benchmarks
dart run benchmark/performance_benchmark.dart

# Run HuggingFace compatibility benchmark
dart run benchmark/hf_compatibility_benchmark.dart

Reproducible HF fixtures

python -m pip install -r scripts/requirements-fixtures.txt
python scripts/generate_hf_fixtures.py

The pinned generator checks original file hashes and verifies that reduced vocabularies reproduce the full-model expected values. Generated goldens are checked in, so ordinary Dart tests do not require Python. The legacy benchmark is a separate diagnostic; the fixture suite is the release compatibility gate.

Hugging Face compatibility in 1.1.0

The suite pins eleven successful model pipelines and two unsupported-pipeline boundaries by repository revision and source-file SHA-256:

Model(s)	What is verified
google-bert/bert-base-uncased, bert-base-cased, bert-base-multilingual-cased	Official JSON pipelines
sentence-transformers/all-MiniLM-L6-v2	Official JSON, including serialized padding
klue/bert-base, klue/roberta-base	Official JSON; Korean normalization and pair type/sequence IDs
google-bert/bert-base-chinese, hfl/chinese-roberta-wwm-ext	Official JSON; Chinese text and differing normalization settings
asafaya/bert-base-arabic, dbmdz/bert-base-turkish-cased, google/muril-base-cased	Official vocabularies with explicitly recorded HF WordPiece builder settings
aubmindlab/bert-base-arabertv02, ai4bharat/IndicBERTv2-MLM-only	Explicit rejection of unsupported AddedToken/Whitespace components

Vocabulary-derived fixtures validate the recorded conversion pipeline; they do not establish full Transformers AutoTokenizer equivalence. Tohoku Japanese v3 and LINE Japanese DistilBERT require MeCab/UniDic preprocessing and are outside this package's verified pipelines. Language coverage does not imply support for every model of that language.

Python tokenizers==0.23.2 generates the checked-in expected values. Tests compare IDs, tokens, type IDs, attention/special masks, offsets, word/sequence IDs and both decoder modes, including sequential and parallel batches. The current suite has 1,143 offline tests and 707 opt-in network tests. CI checks Linux, Windows, Dart 3.10.7 and stable, plus analysis, formatting and publish dry-run. See fixture provenance and regeneration.

Supported JSON components are WordPiece, BertNormalizer, BertPreTokenizer, BertProcessing, WordPiece decoder, and TemplateProcessing with one occurrence of each input sequence and single vocabulary-token special entries. Null normalizer, pre-tokenizer, post-processor and decoder preserve their respective absence. Exact added-token matching (normalized, single_word, lstrip and rstrip all false) is supported. Other component types, unsupported added-token flags and nonzero truncation stride fail explicitly with FormatException. Type IDs come from the template, independently of sequence IDs: KLUE RoBERTa uses type ID zero for both inputs. Type IDs must fit the public Uint8List representation (0–255).

JSON padding and truncation settings are applied, including MiniLM's serialized 128-token padding. Use noPadding()/noTruncation() to disable them. JSON decoder cleanup is honored: for example Hello, world! decodes to hello, world! with the uncased JSON pipeline. The vocab.txt API keeps its legacy spaced decoding (hello , world !). configOverride overrides the exposed WordPiece settings and uses the legacy CLS/SEP configuration instead of the JSON template.

Offsets and character lookup APIs use original Unicode code-point indices, not normalized-text indices or Dart UTF-16 code-unit indices. For example the hello in 😊 hello spans (2, 7). To extract it, use String.fromCharCodes(text.runes.toList().sublist(2, 7)). Word IDs restart at zero for each input sequence; use sequenceIndex for pair word/character lookups. Inserted template and padding tokens have null word/sequence IDs; added-token strings matched within input text retain their input alignment.

Original word IDs are retained during left truncation. This differs from HF 0.23.2's early-truncation optimization in one documented boundary case; the fixture includes both the observed HF output and HF's full-encoding/post-process reference result. This package does not claim universal HF pipeline compatibility.

RUN_HF_NETWORK_TESTS=1 dart test test/huggingface_network_test.dart

Network failures or changed model bytes fail this opt-in suite. The default dart test runs the offline regression suite without downloading models.

PowerShell:

$env:RUN_HF_NETWORK_TESTS = '1'
dart test test/huggingface_network_test.dart
Remove-Item Env:RUN_HF_NETWORK_TESTS

License

The Dart package is MIT licensed. Test fixtures retain their upstream terms, including Apache-2.0 and KLUE CC-BY-SA-4.0; see the fixture attribution. Fixtures and generation tools are excluded from the published package.