core/coop/shard library

Shared data helpers for the coop_* demos:

  • character-level tokeniser and vocab
  • training-window sampler that returns (x, y) sequence pairs
  • built-in self-contained toy corpus so the demos don't require data/tiny_shakespeare.txt

Classes

CharVocab

Constants

kToyCorpus → const String
Small self-contained corpus for the demos. ~1500 chars, exercises enough vocab to train a tiny GPT to visible convergence in seconds.

Functions

sampleWindow(List<double> ids, int blockSize, Random rng, {required Device device}) → (Tensor, Tensor)
Sample one [blockSize] window at a random start position and return the (x, y = next-token) pair as rank-1 tensors on device.
shardSlice(List<double> ids, int shardIndex, int numShards) → List<double>
Partition ids into numShards contiguous, non-overlapping slices and return slice number shardIndex (0-based). Used by the demos to give each worker/replica its own subset of the corpus.