Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification
This work addresses uncertainty quantification for neural network practitioners, offering an incremental improvement over existing methods like Bayesian neural networks.
The paper tackled the problem of uncertainty quantification in neural networks by training infinite mixture models with maximum likelihood instead of variational inference, resulting in increased predictive variance that improved misclassification identification by 15% and robustness against adversarial attacks by 10% compared to standard Bayesian neural networks.
Uncertainty quantification in neural networks gained a lot of attention in the past years. The most popular approaches, Bayesian neural networks (BNNs), Monte Carlo dropout, and deep ensembles have one thing in common: they are all based on some kind of mixture model. While the BNNs build infinite mixture models and are derived via variational inference, the latter two build finite mixtures trained with the maximum likelihood method. In this work we investigate the effect of training an infinite mixture distribution with the maximum likelihood method instead of variational inference. We find that the proposed objective leads to stochastic networks with an increased predictive variance, which improves uncertainty based identification of miss-classification and robustness against adversarial attacks in comparison to a standard BNN with equivalent network structure. The new model also displays higher entropy on out-of-distribution data.