core/optim/adam library
Adam optimizer (Kingma & Ba, 2015) with bias correction.
Update rule (per parameter, per step t, starting at t = 1):
m <- beta1 * m + (1 - beta1) * g
v <- beta2 * v + (1 - beta2) * g * g
m_hat <- m / (1 - beta1^t)
v_hat <- v / (1 - beta2^t)
p <- p - lr * m_hat / (sqrt(v_hat) + eps)
weightDecay > 0 adds a decoupled L2 term (lr * weightDecay * p,
AdamW-style — kept out of the moment estimates).