Variational Bayesian Bow tie Neural Networks with Shrinkage
This work addresses uncertainty estimation and robustness issues in deep learning for practitioners, though it is incremental as it builds on existing Bayesian and variational methods.
The paper tackles the challenges of Bayesian deep learning, such as restrictive variational assumptions and sensitivity to network architecture, by introducing a stochastic relaxation with sparsity-promoting priors, resulting in a fast variational inference algorithm that avoids distributional and independence constraints.
Despite the dominant role of deep models in machine learning, limitations persist, including overconfident predictions, susceptibility to adversarial attacks, and underestimation of variability in predictions. The Bayesian paradigm provides a natural framework to overcome such issues and has become the gold standard for uncertainty estimation with deep models, also providing improved accuracy and a framework for tuning critical hyperparameters. However, exact Bayesian inference is challenging, typically involving variational algorithms that impose strong independence and distributional assumptions. Moreover, existing methods are sensitive to the architectural choice of the network. We address these issues by focusing on a stochastic relaxation of the standard feed-forward rectified neural network and using sparsity-promoting priors on the weights of the neural network for increased robustness to architectural design. Thanks to Polya-Gamma data augmentation tricks, which render a conditionally linear and Gaussian model, we derive a fast, approximate variational inference algorithm that avoids distributional assumptions and independence across layers. Suitable strategies to further improve scalability and account for multimodality are considered.