tokenByteValues method

List<Uint8List> tokenByteValues()

Returns sortedTokenBytes from underlying CoreBPE tokenizer.

Implementation

List<Uint8List> tokenByteValues() => _coreBPE.tokenByteValues();