tokenize property

List<int> Function(String text) tokenize
final

Tokenization function: text → list of token IDs. Injected to decouple model from tokenizer implementation.

Implementation

final List<int> Function(String text) tokenize;