AdamGPU constructor

AdamGPU(
  1. List<GPUTensor> parameters,
  2. double learningRate, {
  3. double beta1 = 0.9,
  4. double beta2 = 0.999,
  5. double epsilon = 1e-7,
  6. double weightDecay = 0.0,
})

Initializes the Adam optimizer and allocates VRAM for the moment caches. Requires the list of trainable parameters and the learningRate. Optional parameters beta1, beta2, epsilon, and weightDecay allow fine-tuning of the algorithm.

Implementation

AdamGPU(
    super.parameters,
    super.learningRate, {
      this.beta1 = 0.9,
      this.beta2 = 0.999,
      this.epsilon = 1e-7,
      this.weightDecay = 0.0,
    }) {
  for (int i = 0; i < parameters.length; i = i + 1) {
    int totalElements = 1;
    for (int s = 0; s < parameters[i].shape.length; s = s + 1) {
      totalElements = totalElements * parameters[i].shape[s];
    }

    GPUTensor<dynamic> m = GPUTensor<dynamic>.empty(parameters[i].shape);
    GPUTensor<dynamic> v = GPUTensor<dynamic>.empty(parameters[i].shape);

    List<double> zeroData = <double>[];
    for (int e = 0; e < totalElements; e = e + 1) {
      zeroData.add(0.0);
    }

    m.pushData(zeroData);
    v.pushData(zeroData);

    mCaches.add(m);
    vCaches.add(v);
  }
}