A PAC-Bayesian Perspective on the Interpolating Information Criterion
It provides a theoretical framework for understanding generalization in overparameterized models, addressing a foundational problem in machine learning theory.
The paper tackles the theory-practice gap in deep learning, particularly benign overfitting, by using the Interpolating Information Criterion to derive a PAC-Bayes bound that quantifies test error dependence on factors like implicit regularization and loss landscape curvature for overparameterized models achieving zero training error.
Deep learning is renowned for its theory-practice gap, whereby principled theory typically fails to provide much beneficial guidance for implementation in practice. This has been highlighted recently by the benign overfitting phenomenon: when neural networks become sufficiently large to interpolate the dataset perfectly, model performance appears to improve with increasing model size, in apparent contradiction with the well-known bias-variance tradeoff. While such phenomena have proven challenging to theoretically study for general models, the recently proposed Interpolating Information Criterion (IIC) provides a valuable theoretical framework to examine performance for overparameterized models. Using the IIC, a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence generalization performance in the interpolating regime. From the provided bound, we quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, optimizer, and parameter-initialization scheme; the spectrum of the empirical neural tangent kernel; curvature of the loss landscape; and noise present in the data.