Conditionally Gaussian PAC-Bayes
This addresses the issue for machine learning researchers and practitioners by providing a more direct optimization method for PAC-Bayesian bounds, though it appears incremental as it builds on prior work in this specific area.
The paper tackled the problem of training stochastic neural networks by optimizing a PAC-Bayesian bound without using surrogate losses, which often misalign with the actual generalization bound, and it resulted in outperforming existing PAC-Bayesian methods in empirical tests.
Recent studies have empirically investigated different methods to train stochastic neural networks on a classification task by optimising a PAC-Bayesian bound via stochastic gradient descent. Most of these procedures need to replace the misclassification error with a surrogate loss, leading to a mismatch between the optimisation objective and the actual generalisation bound. The present paper proposes a novel training algorithm that optimises the PAC-Bayesian bound, without relying on any surrogate loss. Empirical results show that this approach outperforms currently available PAC-Bayesian training methods.