Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimization
This work addresses memory constraints for researchers and practitioners training large models in domains like natural language processing, though it is incremental as it builds on existing methods like Adam.
The paper tackles the problem of memory inefficiency in stochastic optimization for large-scale matrices by proposing Alada, an adaptive momentum method that uses rank-one factorization to estimate gradient second moments, achieving sublinear memory overheads and demonstrating robustness in training large models on NLP tasks.
This work proposes Alada, an adaptive momentum method for stochastic optimization over large-scale matrices. Alada employs a rank-one factorization approach to estimate the second moment of gradients, where factors are updated alternatively to minimize the estimation error. Alada achieves sublinear memory overheads and can be readily extended to optimizing tensor-shaped variables.We also equip Alada with a first moment estimation rule, which enhances the algorithm's robustness without incurring additional memory overheads. The theoretical performance of Alada aligns with that of traditional methods such as Adam. Numerical studies conducted on several natural language processing tasks demonstrate the reduction in memory overheads and the robustness in training large models relative to Adam and its variants.