Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method
This is an incremental improvement for machine learning practitioners dealing with adaptive gradient methods.
The paper tackles the limitation of AdaGrad's monotonically decreasing learning rates by introducing GradaGrad, a method that allows learning rates to increase or decrease adaptively, and demonstrates similar convergence rates and non-monotone adaptation in experiments.
The classical AdaGrad method adapts the learning rate by dividing by the square root of a sum of squared gradients. Because this sum on the denominator is increasing, the method can only decrease step sizes over time, and requires a learning rate scaling hyper-parameter to be carefully tuned. To overcome this restriction, we introduce GradaGrad, a method in the same family that naturally grows or shrinks the learning rate based on a different accumulation in the denominator, one that can both increase and decrease. We show that it obeys a similar convergence rate as AdaGrad and demonstrate its non-monotone adaptation capability with experiments.