bert_tokenizer 1.2.0

Pure Dart WordPiece tokenizer for BERT NLP models. Completely independent of Flutter.

bert_tokenizer

A lightweight, pure Dart WordPiece tokenizer for BERT and other NLP models.

Because this package is entirely independent of Flutter, it can be used anywhere Dart runs: Flutter apps, Dart backend servers (Shelf/Dart Frog), CLI tools, and web applications.

Features

• Pure Dart & Cross-Platform Architecture: Built entirely independently of the Flutter SDK and flutter/services.dart. By accepting vocabulary data as a raw string (fromStringContent), it completely decouples the tokenizer from the file system. This means it runs seamlessly on Dart backend servers (Shelf/Dart Frog), CLI tools, web applications, and Flutter apps.
• Robust WordPiece Tokenization: Goes far beyond simple whitespace splitting. Before applying the WordPiece algorithm, it actively strips invalid control characters, optionally normalizes text to lowercase, and isolates punctuation using regular expressions. It accurately splits compound words into subwords (e.g., worldwide -> world, ##wide) and gracefully falls back to the [UNK] token if an unmappable fragment is encountered.
• Turnkey ML Model Preparation: The prepareNerInput method handles the complex boilerplate of sequence formatting. It automatically truncates oversized inputs to fit your specified maxLength and strictly wraps the token sequence with the standard [CLS] (start) and [SEP] (separator) tokens required by BERT architectures.
• Automatic Padding & Masking: Directly outputs the three aligned integer arrays expected by TFLite and ONNX models. It fills shorter sequences up to the maxLength with [PAD] tokens, generates an attention mask (inputMask) with 1s for real text and 0s for padding so the model knows what to ignore, and initializes segmentIds (token type IDs) with 0s for single-sequence tasks.
• Bidirectional Decoding: Features a convertIdsToTokens utility to translate numerical array predictions from your ML model back into readable string tokens, safely handling out-of-bounds array indices by returning [UNK].

Getting started

Add bert_tokenizer to your pubspec.yaml:

dependencies:
  bert_tokenizer: ^1.0.0

Then run dart pub get or flutter pub get.

Usage

You must provide your own vocabulary file (usually vocab.txt from a pre-trained BERT model) where each line represents a token.

import 'dart:io';
import 'package:bert_tokenizer/bert_tokenizer.dart';

void main() {
  // Load your vocabulary string.
  final vocabContent = File('vocab.txt').readAsStringSync();

  // Initialize the tokenizer
  final tokenizer = BertTokenizer.fromStringContent(vocabContent);

  // Prepare the input for your NER/NLP model
  final text = "Hello worldwide Dart is awesome!";
  final maxLength = 12;

  // Basic usage – all zeros for segment IDs, right truncation
  final input = tokenizer.prepareNerInput(text, maxLength);

  // Feed these arrays directly into your TFLite or ONNX model
  print('Input IDs:    ${input.inputIds}');
  print('Input Mask:   ${input.inputMask}');
  print('Segment IDs:  ${input.segmentIds}');

  //Advanced usage

  // Truncate from the left (keep the last `maxLength-2` tokens)
  final leftTruncated = tokenizer.prepareNerInput(
    text,
    maxLength,
    strategy: TruncationStrategy.left,
  );

  // Set a custom segment ID for all tokens (e.g., for sentence-pair tasks)
  final withSegmentId = tokenizer.prepareNerInput(
    text,
    maxLength,
    segmentId: 1,
  );

  // Provide per‑token segment IDs (length must match token count)
  final tokens = tokenizer.tokenize(text);
  final perTokenSegments = List.generate(tokens.length, (i) => i % 2);
  final customSegments = tokenizer.prepareNerInput(
    text,
    maxLength,
    segmentIds: perTokenSegments,
  );
}

Batch Processing

If you have multiple texts, use prepareBatch for better performance:

final texts = ['Hello world', 'Dart is great'];
final batchInputs = tokenizer.prepareBatch(texts, maxLength);
 for (var input in batchInputs) {
  print(input.inputIds);
}

Pre‑tokenized Input

If you already have a list of tokens (e.g., from a different tokenizer), use prepareFromTokens to skip the tokenization step:

final preTokenized = ['hello', 'world', '##wide'];
final inputFromTokens = tokenizer.prepareFromTokens(preTokenized, maxLength);

Output

The prepareNerInput method returns a BertInput object containing three arrays required by standard BERT models:

• inputIds: The numeric IDs of the tokens in your vocabulary. It automatically prepends the [CLS] (start) token ID, appends the [SEP] (separator) token ID, and fills the rest of the array up to maxLength with the [PAD] (padding) token ID.
• inputMask: Also known as the attention mask. It contains 1 for actual text tokens and 0 for padding tokens, telling the model what to ignore.
• segmentIds: Also known as token type IDs. Used to distinguish between different sentences in sequence-pair tasks. For single-sequence tasks like NER, this will be an array of 0s.

Additional Methods

If you only need to tokenize the text into strings without generating the numeric IDs and masks:

final stringTokens = tokenizer.tokenize("Hello worldwide");
print(stringTokens); // ['hello', 'world', '##wide']

Contributing

Contributions, issues, and feature requests are welcome!