TokenEncoder class abstract

Abstract interface for text tokenizers.

Implementers

• Cl100kEncoder

Constructors

TokenEncoder()

Properties

hashCode → int

The hash code for this object.

no setter, inherited

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

Methods

count(String text) → int

Returns the number of tokens in text.

decode(List<int> tokens) → String

Decodes a list of tokens back into text.

encode(String text) → List<int>

Encodes text into a list of token IDs.

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited