Unified Uncertainty Calibration
This work addresses the need for robust and safe AI systems by improving uncertainty estimation, though it appears incremental as it builds on existing uncertainty calibration methods.
The paper tackles the problem of miscalibrated predictions and poor communication between different uncertainty sources in classifiers by introducing Unified Uncertainty Calibration (U2C), a framework that combines aleatoric and epistemic uncertainties, and it outperforms the reject-or-classify method on various ImageNet benchmarks.
To build robust, fair, and safe AI systems, we would like our classifiers to say ``I don't know'' when facing test examples that are difficult or fall outside of the training classes.The ubiquitous strategy to predict under uncertainty is the simplistic \emph{reject-or-classify} rule: abstain from prediction if epistemic uncertainty is high, classify otherwise.Unfortunately, this recipe does not allow different sources of uncertainty to communicate with each other, produces miscalibrated predictions, and it does not allow to correct for misspecifications in our uncertainty estimates. To address these three issues, we introduce \emph{unified uncertainty calibration (U2C)}, a holistic framework to combine aleatoric and epistemic uncertainties. U2C enables a clean learning-theoretical analysis of uncertainty estimation, and outperforms reject-or-classify across a variety of ImageNet benchmarks. Our code is available at: https://github.com/facebookresearch/UnifiedUncertaintyCalibration