Bayesian Deep Learning via Subnetwork Inference
This work addresses the problem of poor calibration and data inefficiency in deep learning for researchers and practitioners, offering a scalable Bayesian method that is incremental in improving existing approximations.
The paper tackles the challenge of scaling Bayesian inference to large neural networks by proposing subnetwork inference, which focuses on a small subset of weights to compute predictive posteriors, and shows that this method compares favorably to ensembles and other approximations in empirical evaluations.
The Bayesian paradigm has the potential to solve core issues of deep neural networks such as poor calibration and data inefficiency. Alas, scaling Bayesian inference to large weight spaces often requires restrictive approximations. In this work, we show that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors. The other weights are kept as point estimates. This subnetwork inference framework enables us to use expressive, otherwise intractable, posterior approximations over such subsets. In particular, we implement subnetwork linearized Laplace as a simple, scalable Bayesian deep learning method: We first obtain a MAP estimate of all weights and then infer a full-covariance Gaussian posterior over a subnetwork using the linearized Laplace approximation. We propose a subnetwork selection strategy that aims to maximally preserve the model's predictive uncertainty. Empirically, our approach compares favorably to ensembles and less expressive posterior approximations over full networks. Our proposed subnetwork (linearized) Laplace method is implemented within the laplace PyTorch library at https://github.com/AlexImmer/Laplace.