Subspace Inference for Bayesian Deep Learning
This addresses the problem of high-dimensional parameter spaces in Bayesian deep learning for researchers and practitioners, offering a method to improve uncertainty calibration, though it is incremental as it builds on existing subspace and inference techniques.
The paper tackles the challenge of scaling Bayesian inference to deep neural networks by constructing low-dimensional subspaces from SGD trajectories, enabling the application of elliptical slice sampling and variational inference. It shows that Bayesian model averaging in these subspaces produces accurate predictions and well-calibrated uncertainty for regression and image classification.
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we construct low-dimensional subspaces of parameter space, such as the first principal components of the stochastic gradient descent (SGD) trajectory, which contain diverse sets of high performing models. In these subspaces, we are able to apply elliptical slice sampling and variational inference, which struggle in the full parameter space. We show that Bayesian model averaging over the induced posterior in these subspaces produces accurate predictions and well calibrated predictive uncertainty for both regression and image classification.