SpecialTokensSet class

Convenient and type-safe way to represent and manipulate BPE special tokens.

See Tiktoken.encode for more details on usage of special tokens.

Constructors

SpecialTokensSet.all()

const

SpecialTokensSet.custom(Set<String> tokens)

SpecialTokensSet.empty()

const

Properties

hashCode → int

The hash code for this object.

no setter, inherited

isAll → bool

no setter

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

set → Set<String>

no setter

Methods

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited