bert_tokenizer 1.2.0

Pure Dart WordPiece tokenizer for BERT NLP models. Completely independent of Flutter.

1.2.0

• Added batch processing (prepareBatch) and support for pre‑tokenized input (prepareFromTokens).
• Introduced left/right truncation strategy (TruncationStrategy) and custom segment ID override (per‑token or global).
• Exposed special token IDs via getters (padTokenId, unkTokenId, etc.) for easy access.
• Optimised padding with pre‑allocated lists for better performance.
• Preserved leading/trailing whitespace in vocabulary (only strips \r for cross‑platform compatibility).

1.1.2

• Upgraded the Example Code: Replaced the basic example script with a comprehensive, multi-step version that demonstrates raw string tokenization, ML input preparation, and reversing numerical IDs back to text.

1.1.1

• Resolved regex syntax errors by utilizing raw triple quotes for the punctuation character class.

• Enforced strict WordPiece standard (entire word becomes [UNK] if any subword fails).

• Added toLowerCase parameter for cased/uncased model flexibility.

• Added pre-tokenization stripping of invisible control characters.

• Robust line ending handling (\r\n and \n) for cross-platform vocab loading.

1.1.0

• Fixed Truncation Crash: Rewrote the sequence length handling in prepareNerInput to prevent unhandled StateError exceptions when maxLength is set to small values (like 0 or 1).

• Preserved Special Tokens: Adjusted the truncation logic to trim the raw word pieces before injecting the [CLS] and [SEP] tokens, ensuring the structural markers required by BERT models are never overwritten.

• Enforced Minimum Bounds: Added an ArgumentError validation check to prepareNerInput ensuring maxLength is at least 2, which is the mathematical minimum required to contain the mandatory start and end tokens.

• Added Initialization Validation: Updated fromStringContent to verify that both [UNK] and [PAD] are explicitly defined within the parsed vocabulary map, replacing late-stage runtime crashes (!) with a descriptive FormatException during setup.

1.0.2

• Update pub.dev visibility.

1.0.1

• Fix documentation links and update repository visibility.

1.0.0

• Initial release of pure Dart BERT tokenizer.