Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
For researchers and practitioners training large language models, this survey provides a structured overview of optimizer design and benchmarking methodology, highlighting the need for comprehensive evaluation.
This survey reviews optimizer design for large language models, covering classical, adaptive, memory-efficient, second-order, sign-based, low-rank, and matrix-based optimizers. It argues that optimizer research is moving from single-algorithm speedup claims toward rigorous, scale-aware comparisons.
Training large language models requires optimization algorithms that are not only statistically effective, but also computationally and memory efficient at extreme scale. Although Adam remains the dominant optimizer for large-scale language-model pretraining and fine-tuning, recent work has revisited nearly every component of the optimization stack: adaptive moment estimation, decoupled weight decay, memory footprint, curvature approximation, sign-based updates, large-batch stability, low-rank gradient structure, and matrix-wise orthogonalized updates. This survey reviews optimizer design for large language models through a systems-and-optimization lens. We organize the literature into classical first-order optimizers, adaptive optimizers, memory-efficient variants, second-order and curvature-aware methods, sign-based and discovered optimizers, low-rank and projection-based methods, and matrix-based optimizers such as Muon. We also discuss benchmarking methodology, including hyperparameter fairness, scale dependence, wall-clock efficiency, token efficiency, memory overhead, and downstream evaluation. We argue that optimizer research for LLMs is entering a new phase: moving from single-algorithm speedup claims toward rigorous, scale-aware comparisons that jointly evaluate convergence, stability, memory, and implementation complexity.