Implicit Weight Uncertainty in Neural Networks
This addresses the issue of uncertainty estimation in neural networks for applications requiring reliability, though it is incremental as it builds on existing Bayesian deep learning methods.
The paper tackles the problem of neural networks being overconfident and lacking meaningful uncertainty measures on unseen or noisy data by introducing Bayes by Hypernet (BbH), a variational approximation method that uses hypernetworks as implicit distributions, achieving competitive accuracies and predictive uncertainties on MNIST and CIFAR5 tasks while being the most robust against adversarial attacks.
Modern neural networks tend to be overconfident on unseen, noisy or incorrectly labelled data and do not produce meaningful uncertainty measures. Bayesian deep learning aims to address this shortcoming with variational approximations (such as Bayes by Backprop or Multiplicative Normalising Flows). However, current approaches have limitations regarding flexibility and scalability. We introduce Bayes by Hypernet (BbH), a new method of variational approximation that interprets hypernetworks as implicit distributions. It naturally uses neural networks to model arbitrarily complex distributions and scales to modern deep learning architectures. In our experiments, we demonstrate that our method achieves competitive accuracies and predictive uncertainties on MNIST and a CIFAR5 task, while being the most robust against adversarial attacks.