Combining Ensembles and Data Augmentation can Harm your Calibration
This addresses a critical issue for machine learning practitioners who rely on uncertainty estimates, as it identifies a previously overlooked pathology in common techniques.
The paper reveals that combining ensemble methods and data augmentation can harm model calibration, creating a trade-off where improved accuracy comes at the expense of calibration, and proposes a simple correction that achieves state-of-the-art uncertainty calibration on CIFAR-10, CIFAR-100, and ImageNet.
Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model's calibration and robustness. Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness. In this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration. This leads to a trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration. On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy. We investigate this pathology and identify a compounding under-confidence among methods which marginalize over sets of weights and data augmentation techniques which soften labels. Finally, we propose a simple correction, achieving the best of both worlds with significant accuracy and calibration gains over using only ensembles or data augmentation individually. Applying the correction produces new state-of-the art in uncertainty calibration across CIFAR-10, CIFAR-100, and ImageNet.