forward method

List<ValueVector> forward(List<int> idx)

The forward pass for the Transformer Encoder model.

Takes a list of integer token indices idx and returns a list of contextualized ValueVector embeddings.

Implementation

List<ValueVector> forward(List<int> idx) {
  final T = idx.length; // Current sequence length

  if (T > blockSize) {
    throw ArgumentError(
        "Input sequence length ($T) exceeds model's block size ($blockSize). "
        "Consider truncating or padding the input.");
  }

  // 1. Get token and position embeddings and sum them
  // Each input token is converted into an embedding, and positional information is added.
  var x = List.generate(T, (t) {
    final tokEmb = tokenEmbeddings[idx[t]];
    final posEmb = positionEmbeddings[t];
    return tokEmb + posEmb;
  });

  // 2. Pass the sequence through all Transformer Encoder blocks
  for (final block in blocks) {
    x = block.forward(x);
  }

  // 3. Apply final layer normalization to the output of the last block
  x = List.generate(T, (t) => finalLayerNorm.forward(x[t]));

  // The output `x` now contains the contextualized embeddings for each input token.
  return x;
}