Mitigating multiple descents: A model-agnostic framework for risk monotonization
This addresses a fundamental issue in high-dimensional statistics for researchers and practitioners, offering a model-agnostic solution to improve prediction reliability, though it is incremental as it builds on existing cross-validation techniques.
The paper tackles the problem of non-monotonic risk behavior (double/multiple descent) in high-dimensional prediction by developing a cross-validation-based framework that modifies any prediction procedure to achieve monotonic asymptotic risk, with applications to least squares methods and novel oracle risk inequalities.
Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for risk monotonization based on cross-validation that takes as input a generic prediction procedure and returns a modified procedure whose out-of-sample prediction risk is, asymptotically, monotonic in the limiting aspect ratio. As part of our framework, we propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting, respectively, and show that, under very mild assumptions, they provably achieve monotonic asymptotic risk behavior. Our results are applicable to a broad variety of prediction procedures and loss functions, and do not require a well-specified (parametric) model. We exemplify our framework with concrete analyses of the minimum $\ell_2$, $\ell_1$-norm least squares prediction procedures. As one of the ingredients in our analysis, we also derive novel additive and multiplicative forms of oracle risk inequalities for split cross-validation that are of independent interest.