Schattor: Schatten-family methods for deep learning optimization
Provides a theoretical framework and convergence guarantees for adaptive optimization in deep learning, addressing limitations of existing methods.
Schattor proposes a family of adaptive first-order optimizers based on Schatten norms, unifying SGD and Muon, and provides dimension-free stationarity guarantees for stochastic matrix optimization.
Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis. Motivated by the limitations of SGD and the success of adaptive optimizers, we propose {\it Schattor}, a family of adaptive first-order methods based on Schatten norms. Schattor unifies SGD and the recently proposed matrix-variate adaptive optimizer Muon within a single Schatten-norm-based framework. We establish dimension-free stationarity guarantees for methods in the Schattor family for stochastic matrix optimization problems via a novel matrix martingale moment bound. We also develop multi-block extensions that adaptively balance block-wise optimization progress and prove dimension-free stationarity guarantees in this more general setting.