core/optim/sgd library

Stochastic gradient descent with optional heavy-ball momentum.

Update rule (per parameter, when momentum > 0):

v <- momentum * v + grad
p <- p - lr * v

When momentum == 0, the velocity buffer is skipped and the update reduces to p <- p - lr * grad. weightDecay > 0 adds an L2 penalty weightDecay * p to the gradient before the velocity step.

Classes

SGD