Do Bayesian Variational Autoencoders Know What They Don't Know?
This addresses the critical issue of reliable OoD detection for deep neural networks, which is essential for safety in applications like autonomous systems, but the work is incremental as it builds on existing Bayesian methods.
The paper tackles the problem of overconfident Out-of-Distribution (OoD) detection in deep generative models by investigating Bayesian inference methods, such as stochastic gradient MCMC and Bayes by Backpropagation, applied to Variational Autoencoders, and introduces two simple scores that achieve state-of-the-art performance on OoD benchmarks.
The problem of detecting the Out-of-Distribution (OoD) inputs is of paramount importance for Deep Neural Networks. It has been previously shown that even Deep Generative Models that allow estimating the density of the inputs may not be reliable and often tend to make over-confident predictions for OoDs, assigning to them a higher density than to the in-distribution data. This over-confidence in a single model can be potentially mitigated with Bayesian inference over the model parameters that take into account epistemic uncertainty. This paper investigates three approaches to Bayesian inference: stochastic gradient Markov chain Monte Carlo, Bayes by Backpropagation, and Stochastic Weight Averaging-Gaussian. The inference is implemented over the weights of the deep neural networks that parameterize the likelihood of the Variational Autoencoder. We empirically evaluate the approaches against several benchmarks that are often used for OoD detection: estimation of the marginal likelihood utilizing sampled model ensemble, typicality test, disagreement score, and Watanabe-Akaike Information Criterion. Finally, we introduce two simple scores that demonstrate the state-of-the-art performance.