encodeOrdinary method

Uint32List encodeOrdinary(
  1. String text
)

Encodes a string into tokens, ignoring special tokens.

This is equivalent to encode(text, disallowedSpecial = SpecialTokensSet.empty()) (but slightly faster).

Example:

final enc = getEncoding("gpt2");
enc.encode_ordinary("hello world"); // [31373, 995]

Implementation

Uint32List encodeOrdinary(String text) {
  return _coreBPE.encodeOrdinaryNative(text);
}