PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
This addresses the issue of model misspecification in Bayesian inference, which can degrade predictive accuracy, though it appears incremental as it builds on existing PAC-Bayes frameworks.
The paper tackles the problem of poor generalization in Bayesian models under misspecification by developing a multi-sample loss (PAC^m) that narrows the gap between inferential and predictive risks, with empirical results showing improved predictive distribution performance.
The Bayesian posterior minimizes the "inferential risk" which itself bounds the "predictive risk". This bound is tight when the likelihood and prior are well-specified. However since misspecification induces a gap, the Bayesian posterior predictive distribution may have poor generalization performance. This work develops a multi-sample loss (PAC$^m$) which can close the gap by spanning a trade-off between the two risks. The loss is computationally favorable and offers PAC generalization guarantees. Empirical study demonstrates improvement to the predictive distribution.