A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks
This provides theoretical guarantees for efficient neural network optimization, particularly in ill-conditioned problems, which is incremental but addresses a known bottleneck in training.
The paper tackled the convergence of Gauss-Newton methods for training neural networks, proving exponential last-iterate convergence to optimal predictors in underparameterized regimes without regularization and showing fast convergence in overparameterized regimes with damping.
We analyze the convergence of Gauss-Newton dynamics for training neural networks with smooth activation functions. In the underparameterized regime, the Gauss-Newton gradient flow induces a Riemannian gradient flow on a low-dimensional, smooth, embedded submanifold of the Euclidean output space. Using tools from Riemannian optimization, we prove \emph{last-iterate} convergence of the Riemannian gradient flow to the optimal in-class predictor at an \emph{exponential rate} that is independent of the conditioning of the Gram matrix, \emph{without} requiring explicit regularization. We further characterize the critical impacts of the neural network scaling factor and the initialization on the convergence behavior. In the overparameterized regime, we show that the Levenberg-Marquardt dynamics with an appropriately chosen damping schedule yields fast convergence rate despite potentially ill-conditioned neural tangent kernel matrices, analogous to the underparameterized regime. These findings demonstrate the potential of Gauss-Newton methods for efficiently optimizing neural networks in the near-initialization regime, particularly in ill-conditioned problems where kernel and Gram matrices have small singular values.