encodeOrdinary method
Encodes a string into tokens, ignoring special tokens.
This is equivalent to encode(text, disallowedSpecial = SpecialTokensSet.empty())
(but slightly faster).
Example:
final enc = getEncoding("gpt2");
enc.encode_ordinary("hello world"); // [31373, 995]
Implementation
Uint32List encodeOrdinary(String text) {
return _coreBPE.encodeOrdinaryNative(text);
}