tokenize method

@override Future<List<int>> tokenize(String text)

The tokens of text, special tokens added as generate adds them (the vocabulary decides about BOS).

Implementation

@override
Future<List<int>> tokenize(String text) async {
  _checkOpen();
  return _queue.run(() async => List<int>.generate(text.length ~/ 4, (i) => i));
}