Minimal penalties and the slope heuristics: a survey
It addresses the problem of model selection in statistics, offering a survey with incremental improvements and new insights for researchers in this domain.
This paper reviews the slope heuristics method for selecting optimal penalties in model selection, presenting theoretical results, self-contained proofs, and new contributions such as connections to residual-variance estimators and practical algorithms tested on synthetic data.
Birg{é} and Massart proposed in 2001 the slope heuristics as a way to choose optimally from data an unknown multiplicative constant in front of a penalty. It is built upon the notion of minimal penalty, and it has been generalized since to some "minimal-penalty algorithms". This paper reviews the theoretical results obtained for such algorithms, with a self-contained proof in the simplest framework, precise proof ideas for further generalizations, and a few new results. Explicit connections are made with residual-variance estimators-with an original contribution on this topic, showing that for this task the slope heuristics performs almost as well as a residual-based estimator with the best model choice-and some classical algorithms such as L-curve or elbow heuristics, Mallows' C p , and Akaike's FPE. Practical issues are also addressed, including two new practical definitions of minimal-penalty algorithms that are compared on synthetic data to previously-proposed definitions. Finally, several conjectures and open problems are suggested as future research directions.