Measuring Uncertainty through Bayesian Learning of Deep Neural Network Structure
This work addresses uncertainty quantification in deep learning for researchers and practitioners, offering an incremental improvement by applying Bayesian principles to network structure rather than weights.
The paper tackles the challenge of Bayesian inference in high-dimensional neural networks by performing Bayesian learning on network structure, drawing inspiration from neural architecture search and developing an efficient stochastic variational inference approach. The method achieves competitive predictive performance while preserving Bayesian uncertainty quantification benefits across challenging scenarios.
Bayesian neural networks (BNNs) augment deep networks with uncertainty quantification by Bayesian treatment of the network weights. However, such models face the challenge of Bayesian inference in a high-dimensional and usually over-parameterized space. This paper investigates a new line of Bayesian deep learning by performing Bayesian inference on network structure. Instead of building structure from scratch inefficiently, we draw inspirations from neural architecture search to represent the network structure. We then develop an efficient stochastic variational inference approach which unifies the learning of both network structure and weights. Empirically, our method exhibits competitive predictive performance while preserving the benefits of Bayesian principles across challenging scenarios. We also provide convincing experimental justification for our modeling choice.