CoreBPE class

BPE tokeniser

Read more on Wikipedia

Constructors

CoreBPE.create(Map<ByteArray, int> encoder, Map<String, int> specialTokensEncoder, String pattern)

factory

Properties

decoder → HashMap<int, ByteArray>

final

encoder → HashMap<ByteArray, int>

final

hashCode → int

The hash code for this object.

no setter, inherited

regex → RegExp

final

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

sortedTokenBytes → List<ByteArray>

final

specialRegex → RegExp

final

specialTokensDecoder → HashMap<int, ByteArray>

final

specialTokensEncoder → HashMap<String, int>

final

Methods

decodeNative(List<int> tokens) → ByteArray

decodeSingleTokenBytes(int token) → ByteArray

encodeNative(String text, Set<String> allowedSpecial) → _Tuple2<Uint32List, int>

encodeOrdinaryNative(String text) → Uint32List

encodeSingleToken(ByteArray bytes) → int

encodeUnstableNative(String text, Set<String> allowedSpecial) → _Tuple2<List<int>, Set<List<int>>>

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

tokenByteValues() → List<Uint8List>

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited