Generalization for Least Squares Regression With Simple Spiked Covariances
This work addresses a gap in analyzing generalization for neural networks, though it is incremental as it builds on prior random matrix theory results.
The paper tackles the problem of understanding generalization error for linear models with spiked covariances, deriving the error in asymptotic regimes and showing that spike eigenvectors and eigenvalues significantly affect it.
Random matrix theory has proven to be a valuable tool in analyzing the generalization of linear models. However, the generalization properties of even two-layer neural networks trained by gradient descent remain poorly understood. To understand the generalization performance of such networks, it is crucial to characterize the spectrum of the feature matrix at the hidden layer. Recent work has made progress in this direction by describing the spectrum after a single gradient step, revealing a spiked covariance structure. Yet, the generalization error for linear models with spiked covariances has not been previously determined. This paper addresses this gap by examining two simple models exhibiting spiked covariances. We derive their generalization error in the asymptotic proportional regime. Our analysis demonstrates that the eigenvector and eigenvalue corresponding to the spike significantly influence the generalization error.