Two models of double descent for weak features
This provides theoretical insights into overparameterization effects in machine learning, though it is incremental as it builds on existing double descent concepts.
The paper mathematically analyzes the double descent risk curve in two simple data models using least squares/least norm predictors, showing that risk peaks when the number of features is near the sample size and decreases beyond that point, contrasting with prescient models that select features optimally.
The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifically, it is shown that the risk peaks when the number of features $p$ is close to the sample size $n$, but also that the risk decreases towards its minimum as $p$ increases beyond $n$. This behavior is contrasted with that of "prescient" models that select features in an a priori optimal order.