Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
This work addresses optimization bottlenecks for deep learning practitioners by providing a unified theoretical framework that reveals connections between existing methods and enables new algorithm development.
The paper tackles the fundamental trade-off between adapting to problem geometry and leveraging curvature information in deep learning optimization by proposing a unified framework based on preconditioned matrix norms that generalizes steepest descent, quasi-Newton, and adaptive methods. The result includes two new optimizers, MuAdam and MuAdam-SANIA, which are shown to be competitive with or outperform existing state-of-the-art methods in experiments.
Optimization lies at the core of modern deep learning, yet existing methods often face a fundamental trade-off between adapting to problem geometry and leveraging curvature utilization. Steepest descent algorithms adapt to different geometries through norm choices but remain strictly first-order, whereas quasi-Newton and adaptive optimizers incorporate curvature information but are restricted to Frobenius geometry, limiting their applicability across diverse architectures. In this work, we propose a unified framework generalizing steepest descent, quasi-Newton methods, and adaptive methods through the novel notion of preconditioned matrix norms. This abstraction reveals that widely used optimizers such as SGD and Adam, as well as more advanced approaches like Muon and KL-Shampoo, and recent hybrids including SOAP and SPlus, all emerge as special cases of the same principle. Within this framework, we provide the first systematic treatment of affine and scale invariance in the matrix-parameterized setting, establishing necessary and sufficient conditions under generalized norms. Building on this foundation, we introduce two new methods, $\texttt{MuAdam}$ and $\texttt{MuAdam-SANIA}$, which combine the spectral geometry of Muon with Adam-style preconditioning. Our experiments demonstrate that these optimizers are competitive with, and in some cases outperform, existing state-of-the-art methods. Our code is available at https://github.com/brain-lab-research/LIB/tree/quasi_descent