encodeOrdinary method

Uint32List encodeOrdinary(
  1. String text
)

Encodes a string into tokens, ignoring special tokens.

This is equivalent to encode(text, disallowedSpecial = SpecialTokensSet.empty()) (but slightly faster).

Example:

final enc = Tiktoken.getEncoderForModel(OpenAiModel.gpt_4);
enc.encodeOrdinary("hello world");

Implementation

Uint32List encodeOrdinary(String text) => _coreBPE.encodeOrdinaryNative(text);