Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
This addresses the memory bottleneck in training large neural networks, making adaptive optimization more scalable, though it is an incremental improvement over existing methods.
The paper tackles the high memory cost of adaptive learning rate methods like Adam by proposing Adafactor, which reduces memory usage by maintaining only per-row and per-column sums of second-moment estimators instead of per-parameter ones, achieving comparable results to Adam on the WMT 2014 English-German translation task with minimal auxiliary storage.
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.