AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations
This addresses a key limitation in optimization for deep learning practitioners, though it is incremental as it builds on existing adaptive methods.
The paper tackles the poor generalization of adaptive optimization methods in deep learning by proposing AdaL, which transforms gradients to balance early acceleration and later stabilization, resulting in improved convergence and generalization on benchmarks.
Adaptive optimization methods have been widely used in deep learning. They scale the learning rates adaptively according to the past gradient, which has been shown to be effective to accelerate the convergence. However, they suffer from poor generalization performance compared with SGD. Recent studies point that smoothing exponential gradient noise leads to generalization degeneration phenomenon. Inspired by this, we propose AdaL, with a transformation on the original gradient. AdaL accelerates the convergence by amplifying the gradient in the early stage, as well as dampens the oscillation and stabilizes the optimization by shrinking the gradient later. Such modification alleviates the smoothness of gradient noise, which produces better generalization performance. We have theoretically proved the convergence of AdaL and demonstrated its effectiveness on several benchmarks.