Least Squares Regression Can Exhibit Under-Parameterized Double Descent
This work addresses a foundational issue in machine learning theory, challenging established assumptions about model generalization and bias-variance trade-offs, which could impact researchers and practitioners in AI and statistics.
The paper tackles the problem of understanding double descent in machine learning by showing that it can occur in the under-parameterized regime, contrary to prior beliefs that it only happens in over-parameterized settings, through two simple examples that provably demonstrate this phenomenon.
The relationship between the number of training data points, the number of parameters, and the generalization capabilities of models has been widely studied. Previous work has shown that double descent can occur in the over-parameterized regime and that the standard bias-variance trade-off holds in the under-parameterized regime. These works provide multiple reasons for the existence of the peak. We postulate that the location of the peak depends on the technical properties of both the spectrum as well as the eigenvectors of the sample covariance. We present two simple examples that provably exhibit double descent in the under-parameterized regime and do not seem to occur for reasons provided in prior work.