Riemannian Lyapunov Optimizer: A Unified Framework for Optimization
This work provides a systematic toolkit for designing stable optimizers, bridging control theory and machine learning, but it appears incremental as it builds on existing optimization concepts.
The authors tackled the problem of unifying optimization algorithms by introducing Riemannian Lyapunov Optimizers (RLOs), a framework derived from control theory that recovers classic methods and achieves state-of-the-art performance in large-scale benchmarks.
We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework. Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. We formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold. This perspective yields a constructive ``optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs. We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks. Overall, RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit for designing stable, effective optimizers.