PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory
Provides a new theoretical framework for finite-sample generalization guarantees in overparameterized and singular models, addressing a key bottleneck in modern machine learning.
The authors derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors using singular learning theory, yielding tighter bounds than classical complexity-based methods for overparameterized models. Applications to low-rank matrix completion and ReLU neural networks show substantial improvements.
We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers, which require explicit control of the model space in terms of metric entropy (integrals), our analysis yields posterior-averaged risk bounds that can be applied to overparameterized models and adapt to the data structure and the intrinsic model complexity. The bound involves a marginal-type integral over the parameter space, which we analyze using tools from singular learning theory to obtain explicit and practically meaningful characterizations of the posterior risk. Applications to low-rank matrix completion and ReLU neural network regression and classification show that the resulting bounds are analytically tractable and substantially tighter than classical complexity-based bounds. Our results highlight the potential of PAC-Bayes analysis for precise finite-sample generalization guarantees in modern overparameterized and singular models.