Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support
This addresses a specific issue in probabilistic programming for researchers and practitioners, offering incremental improvements to prediction robustness.
The paper tackles the problem of unstable Bayesian model averaging (BMA) weights in probabilistic programs with stochastic support, which can lead to sub-optimal predictions. They propose two alternative path-weighting mechanisms based on stacking and PAC-Bayes, showing in experiments that these methods are more robust and yield better predictions compared to default BMA.
The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.