Gibbs-Based Information Criteria and the Over-Parameterized Regime
This work addresses a foundational problem in machine learning theory for researchers studying model selection and generalization in over-parameterized settings, though it is incremental as it builds on existing information-theoretic frameworks.
The paper tackles the mismatch between classical information criteria and the double-descent phenomenon in over-parameterized models by developing Gibbs-based AIC and BIC, linking penalties to information measures like symmetrized KL information and KL divergence. It shows that Gibbs-based BIC can select high-dimensional models and reveals discrepancies between marginal likelihood and population risk in over-parameterized regimes.
Double-descent refers to the unexpected drop in test loss of a learning algorithm beyond an interpolating threshold with over-parameterization, which is not predicted by information criteria in their classical forms due to the limitations in the standard asymptotic approach. We update these analyses using the information risk minimization framework and provide Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) for models learned by the Gibbs algorithm. Notably, the penalty terms for the Gibbs-based AIC and BIC correspond to specific information measures, i.e., symmetrized KL information and KL divergence. We extend this information-theoretic analysis to over-parameterized models by providing two different Gibbs-based BICs to compute the marginal likelihood of random feature models in the regime where the number of parameters $p$ and the number of samples $n$ tend to infinity, with $p/n$ fixed. Our experiments demonstrate that the Gibbs-based BIC can select the high-dimensional model and reveal the mismatch between marginal likelihood and population risk in the over-parameterized regime, providing new insights to understand double-descent.