Gibbs Sampling the Posterior of Neural Networks
This work addresses the challenge of posterior sampling in neural networks for researchers in Bayesian machine learning, though it is incremental as it builds on existing MCMC methods with a new model.
The paper tackles the problem of sampling from the posterior of neural networks by proposing a new probabilistic model with added noise, enabling efficient Gibbs sampling that achieves performance similar to state-of-the-art MCMC methods like HMC and MALA on small models with real and synthetic data.
In this paper, we study sampling from a posterior derived from a neural network. We propose a new probabilistic model consisting of adding noise at every pre- and post-activation in the network, arguing that the resulting posterior can be sampled using an efficient Gibbs sampler. For small models, the Gibbs sampler attains similar performances as the state-of-the-art Markov chain Monte Carlo (MCMC) methods, such as the Hamiltonian Monte Carlo (HMC) or the Metropolis adjusted Langevin algorithm (MALA), both on real and synthetic data. By framing our analysis in the teacher-student setting, we introduce a thermalization criterion that allows us to detect when an algorithm, when run on data with synthetic labels, fails to sample from the posterior. The criterion is based on the fact that in the teacher-student setting we can initialize an algorithm directly at equilibrium.