cpu_version/optimizers/optimizer library
Holds and exports all optimization algorithms used for training models.
Classes
- Adagrad
- Implements the Adagrad optimization algorithm. Adapts the learning rate individually for each parameter based on the accumulated history of squared gradients.
- Adam
- Implements the Adam (Adaptive Moment Estimation) optimization algorithm. Computes individual adaptive learning rates from exponentially decaying averages of past gradients (first moment) and squared gradients (second moment), incorporating bias correction terms.
- AdamW
- Implements the AdamW optimization algorithm with decoupled weight decay. Modifies the standard Adam update by applying weight decay directly to the parameters rather than incorporating it into the gradient moments.
- AMSGrad
-
Implements the AMSGrad optimization algorithm, a variant of Adam.
Maintains the maximum of past squared gradient averages in
_vHatto prevent learning rate scaling from exploding, ensuring non-increasing effective step sizes for improved convergence. - Momentum
- Implements the Stochastic Gradient Descent with Momentum optimization algorithm. Accelerates gradient vectors in the relevant direction by accumulating past velocity updates, dampening oscillations across optimization steps.
- NAG
- Implements Nesterov Accelerated Gradient (NAG) optimization. Incorporates momentum by computing the parameter update step using a look-ahead velocity term.
- Optimizer
- The abstract base class for all optimization algorithms. Tracks a collection of trainable parameters and manages optimization steps and gradient resets.
- RMSprop
- Implements the RMSprop optimization algorithm. Maintains a moving average of squared gradients to normalize gradient magnitudes, dividing the learning rate by the square root of recent gradient variances.
- SGD
- Implements the Stochastic Gradient Descent (SGD) optimization algorithm. Updates parameters in the opposite direction of the gradient scaled directly by learningRate.