TextTokenDataset class

Inheritance

Constructors

TextTokenDataset.fromFile(String path, {required dynamic tokenizer, required int blockSize, Device device = Device.CPU})
Tokenize the contents of a UTF-8 text file. Convenience wrapper over TextTokenDataset.fromText.
factory
TextTokenDataset.fromText(String text, {required dynamic tokenizer, required int blockSize, Device device = Device.CPU})
Tokenize text and expose sliding-window LM examples.
factory
TextTokenDataset.fromTokens(List<int> tokens, {required int blockSize, Device device = Device.CPU})
Build directly from a pre-tokenized id list — bypasses the tokenizer requirement, useful when tokens live in a compact binary file.
factory

Properties

blockSize → int
final
device → Device
final
hashCode → int
The hash code for this object.
no setterinherited
length → int
no setteroverride
numTokens → int
no setter
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
tokens → List<int>
final

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited
operator [](int index) → LmSample
override