CharTokenizer.fromText constructor

CharTokenizer.fromText(
  1. String text
)

Fit a tokenizer to text.

Vocabulary is the set of characters in text, sorted for determinism (so the same text always yields the same id map across runs and machines).

Implementation

factory CharTokenizer.fromText(String text) {
  final chars = text.split('').toSet().toList()..sort();
  final stoi = <String, int>{
    for (int i = 0; i < chars.length; i++) chars[i]: i,
  };
  return CharTokenizer._(chars, stoi);
}