Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks
This work addresses the implicit bias in neural network optimization for researchers in machine learning theory, providing incremental insights into spectral bias and dynamics across parameterization regimes.
The paper analyzes the dynamics of neural networks optimizing mean squared error via gradient flow, showing that in the underparameterized regime, networks learn eigenfunctions of an integral operator related to the Neural Tangent Kernel at rates tied to eigenvalues, such as spherical harmonics for uniformly distributed data on a sphere. It introduces 'Damped Deviations' as a unifying perspective to track empirical risk dynamics, extending results to overparameterized settings.
We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that in the underparameterized regime the network learns eigenfunctions of an integral operator $T_{K^\infty}$ determined by the Neural Tangent Kernel (NTK) at rates corresponding to their eigenvalues. For example, for uniformly distributed data on the sphere $S^{d - 1}$ and rotation invariant weight distributions, the eigenfunctions of $T_{K^\infty}$ are the spherical harmonics. Our results can be understood as describing a spectral bias in the underparameterized regime. The proofs use the concept of "Damped Deviations", where deviations of the NTK matter less for eigendirections with large eigenvalues due to the occurence of a damping factor. Aside from the underparameterized regime, the damped deviations point-of-view can be used to track the dynamics of the empirical risk in the overparameterized setting, allowing us to extend certain results in the literature. We conclude that damped deviations offers a simple and unifying perspective of the dynamics when optimizing the squared error.