On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks
This work addresses the problem of improving predictive calibration in deep learning for researchers, showing incremental results that challenge the utility of mode exploration.
The paper investigated whether exploring local regions around diverse solutions (posterior modes) improves Bayesian model averaging for neural networks, finding that simple mode exploration methods on CIFAR-10 produced little to no improvement over ensembles without such exploration.
Multiple techniques for producing calibrated predictive probabilities using deep neural networks in supervised learning settings have emerged that leverage approaches to ensemble diverse solutions discovered during cyclic training or training from multiple random starting points (deep ensembles). However, only a limited amount of work has investigated the utility of exploring the local region around each diverse solution (posterior mode). Using three well-known deep architectures on the CIFAR-10 dataset, we evaluate several simple methods for exploring local regions of the weight space with respect to Brier score, accuracy, and expected calibration error. We consider both Bayesian inference techniques (variational inference and Hamiltonian Monte Carlo applied to the softmax output layer) as well as utilizing the stochastic gradient descent trajectory near optima. While adding separate modes to the ensemble uniformly improves performance, we show that the simple mode exploration methods considered here produce little to no improvement over ensembles without mode exploration.