SPAiSimpleBpeTokenizer class
A class that implements a simple BPE (Byte Pair Encoding) tokenizer.
Constructors
- SPAiSimpleBpeTokenizer()
-
A factory constructor that returns a singleton instance of SPGPTTokenizer.
If an instance has already been created, it returns the existing instance. Otherwise, it creates a new instance of SPGPTTokenizer.
factory
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
encodeString(
String stringToEncode) → Future< SPTokenContainer> - A private member variable for caching a list of BPE (Byte Pair Encoding) merges.
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited