Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
This addresses a robustness issue in stochastic optimization for machine learning practitioners, offering an incremental improvement over existing adaptive methods.
The authors tackled the problem of adaptive gradient methods like Adam diverging on certain convex optimization problems by proposing Expectigrad, which uses a per-component unweighted mean of historical gradients and a bias-corrected momentum term, proving it cannot diverge on instances where Adam does and showing it performs favorably on high-dimensional tasks with little tuning.
Many popular adaptive gradient methods such as Adam and RMSProp rely on an exponential moving average (EMA) to normalize their stepsizes. While the EMA makes these methods highly responsive to new gradient information, recent research has shown that it also causes divergence on at least one convex optimization problem. We propose a novel method called Expectigrad, which adjusts stepsizes according to a per-component unweighted mean of all historical gradients and computes a bias-corrected momentum term jointly between the numerator and denominator. We prove that Expectigrad cannot diverge on every instance of the optimization problem known to cause Adam to diverge. We also establish a regret bound in the general stochastic nonconvex setting that suggests Expectigrad is less susceptible to gradient variance than existing methods are. Testing Expectigrad on several high-dimensional machine learning tasks, we find it often performs favorably to state-of-the-art methods with little hyperparameter tuning.