Analysis of Interpolating Regression Models and the Double Descent Phenomenon
This addresses a fundamental problem in machine learning theory by challenging classical bias-variance tradeoff assumptions, providing insights for researchers and practitioners working with overparametrized models.
The paper tackles the counterintuitive generalization of overparametrized regression models that interpolate noisy training data, analyzing the double descent phenomenon where test error decreases after an initial peak as model complexity increases. It explains the peak location and shape of the test error curve based on the smallest singular value of the regression matrix.
A regression model with more parameters than data points in the training data is overparametrized and has the capability to interpolate the training data. Based on the classical bias-variance tradeoff expressions, it is commonly assumed that models which interpolate noisy training data are poor to generalize. In some cases, this is not true. The best models obtained are overparametrized and the testing error exhibits the double descent behavior as the model order increases. In this contribution, we provide some analysis to explain the double descent phenomenon, first reported in the machine learning literature. We focus on interpolating models derived from the minimum norm solution to the classical least-squares problem and also briefly discuss model fitting using ridge regression. We derive a result based on the behavior of the smallest singular value of the regression matrix that explains the peak location and the double descent shape of the testing error as a function of model order.