Approximate blocked Gibbs sampling for Bayesian neural networks
This work addresses a computational bottleneck for researchers and practitioners using Bayesian neural networks, offering an incremental improvement in sampling efficiency.
The paper tackles the challenge of making minibatch MCMC sampling feasible for Bayesian neural networks by proposing a blocked Gibbs sampling scheme that partitions parameters to handle varying layer widths and adjusts proposal variances to prevent vanishing acceptance rates in deep networks, resulting in improved predictive accuracy in classification tasks and enabling uncertainty quantification.
In this work, minibatch MCMC sampling for feedforward neural networks is made more feasible. To this end, it is proposed to sample subgroups of parameters via a blocked Gibbs sampling scheme. By partitioning the parameter space, sampling is possible irrespective of layer width. It is also possible to alleviate vanishing acceptance rates for increasing depth by reducing the proposal variance in deeper layers. Increasing the length of a non-convergent chain increases the predictive accuracy in classification tasks, so avoiding vanishing acceptance rates and consequently enabling longer chain runs have practical benefits. Moreover, non-convergent chain realizations aid in the quantification of predictive uncertainty. An open problem is how to perform minibatch MCMC sampling for feedforward neural networks in the presence of augmented data.