layerNormMatrixGPU function

GPUTensor<Matrix> layerNormMatrixGPU(GPUTensor<Matrix> m, GPUTensor<Vector> gamma, GPUTensor<Vector> beta, GPUTensor<Vector> meanCache, GPUTensor<Vector> rstdCache, double epsilon, CommandBuffer tape, { GPUTensor<Matrix>? outTensor, })

Normalizes a 2D matrix independently for each row. Calculates the mean and variance across the columns of a single row, normalizes the row's elements, and applies the gamma and beta transformations. Outputs a normalized matrix of the original dimensions. The operations are appended to tape for execution.

Implementation

GPUTensor<Matrix> layerNormMatrixGPU(
    GPUTensor<Matrix> m,
    GPUTensor<Vector> gamma,
    GPUTensor<Vector> beta,
    GPUTensor<Vector> meanCache,
    GPUTensor<Vector> rstdCache,
    double epsilon,
    CommandBuffer tape,
    {GPUTensor<Matrix>? outTensor}) {
  int numRows = m.shape[0];
  int numCols = m.shape[1];

  GPUTensor<Matrix> out =
      outTensor ?? GPUTensor<Matrix>.empty(<int>[numRows, numCols]);

  tape.putInt(OP_LAYER_NORM_FORWARD);
  tape.putString(m.id);
  tape.putString(gamma.id);
  tape.putString(beta.id);
  tape.putString(out.id);
  tape.putString(meanCache.id);
  tape.putString(rstdCache.id);
  tape.putFloat(epsilon);

  out.creator = GPUNode(
    [m, gamma, beta],
    (CommandBuffer bTape) {
      bTape.putInt(OP_LAYER_NORM_BACKWARD);
      bTape.putString(out.gradId);
      bTape.putString(m.id);
      bTape.putString(gamma.id);
      bTape.putString(meanCache.id);
      bTape.putString(rstdCache.id);
      bTape.putString(m.gradId);
      bTape.putString(gamma.gradId);
      bTape.putString(beta.gradId);
    },
    opName: 'layerNormMatrixGPU',
    cost: numRows * numCols * 8,
  );

  return out;
}