PAC-Bayes with Backprop
This work addresses the challenge of certifying neural network risk without data splitting, potentially benefiting machine learning practitioners seeking reliable uncertainty estimates, though it appears incremental with novel bounds.
The paper tackles the problem of training probabilistic neural networks by minimizing PAC-Bayes bounds to achieve competitive test errors and non-vacuous risk certifications, with results showing ~1.4% test error and ~2.3% bound on MNIST.
We explore the family of methods "PAC-Bayes with Backprop" (PBB) to train probabilistic neural networks by minimizing PAC-Bayes bounds. We present two training objectives, one derived from a previously known PAC-Bayes bound, and a second one derived from a novel PAC-Bayes bound. Both training objectives are evaluated on MNIST and on various UCI data sets. Our experiments show two striking observations: we obtain competitive test set error estimates (~1.4% on MNIST) and at the same time we compute non-vacuous bounds with much tighter values (~2.3% on MNIST) than previous results. These observations suggest that neural nets trained by PBB may lead to self-bounding learning, where the available data can be used to simultaneously learn a predictor and certify its risk, with no need to follow a data-splitting protocol.