Quantifying Overfitting along the Regularization Path for Two-Part-Code MDL in Supervised Classification
This work addresses theoretical understanding of regularization and overfitting in MDL-based classification, offering incremental improvements to prior analysis.
The paper tackles the problem of quantifying overfitting in a modified two-part-code Minimum Description Length (MDL) learning rule for binary classification by characterizing the entire regularization curve, providing a precise quantitative description of worst-case limiting error as a function of regularization parameter and noise level.
We provide a complete characterization of the entire regularization curve of a modified two-part-code Minimum Description Length (MDL) learning rule for binary classification, based on an arbitrary prior or description language. Grunwald and Langford [2004] previously established the lack of asymptotic consistency, from an agnostic PAC (frequentist worst case) perspective, of the MDL rule with a penalty parameter of $λ=1$, suggesting that it underegularizes. Driven by interest in understanding how benign or catastrophic under-regularization and overfitting might be, we obtain a precise quantitative description of the worst case limiting error as a function of the regularization parameter $λ$ and noise level (or approximation error), significantly tightening the analysis of Grunwald and Langford for $λ=1$ and extending it to all other choices of $λ$.