Greedy Bayesian Posterior Approximation with Deep Ensembles
This work addresses a limitation in uncertainty estimation for deep learning, particularly in computer vision, by providing a more controlled and theoretically grounded approach to ensemble training, though it is incremental as it builds on existing ensemble methods.
The paper tackles the uncontrolled approximation of posterior distributions in deep ensembles by proposing a principled method that minimizes an f-divergence between the true posterior and a kernel density estimator, demonstrating improved performance on computer vision out-of-distribution detection benchmarks across multiple datasets and architectures.
Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixture of delta functions. The training of ensembles relies on non-convexity of the loss landscape and random initialization of their individual members, making the resulting posterior approximation uncontrolled. This paper proposes a novel and principled method to tackle this limitation, minimizing an $f$-divergence between the true posterior and a kernel density estimator (KDE) in a function space. We analyze this objective from a combinatorial point of view, and show that it is submodular with respect to mixture components for any $f$. Subsequently, we consider the problem of greedy ensemble construction. From the marginal gain on the negative $f$-divergence, which quantifies an improvement in posterior approximation yielded by adding a new component into the KDE, we derive a novel diversity term for ensemble methods. The performance of our approach is demonstrated on computer vision out-of-distribution detection benchmarks in a range of architectures trained on multiple datasets. The source code of our method is made publicly available at https://github.com/Oulu-IMEDS/greedy_ensembles_training.