Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty
This addresses the challenge of scalable uncertainty quantification in BNNs for researchers and practitioners, though it is incremental as it builds on existing variational inference techniques.
The paper tackles the problem of combining structural and parameter uncertainty in Bayesian neural networks (BNNs) by applying model uncertainty for structural learning and adapting scalable variational inference with reparametrization. The result is a method that achieves comparable accuracy to competing models while being much more sparse than ordinary BNNs.
Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: Parameter and prediction uncertainties become easily available, facilitating rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, so far, there have been no scalable techniques capable of combining both structural and parameter uncertainty. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and hence make inference in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs.