Evaluating and Boosting Uncertainty Quantification in Classification
This work addresses the need for reliable uncertainty quantification in machine-assisted medical decision-making, though it is incremental in nature.
The authors tackled the lack of a unified evaluation metric for uncertainty quantification in classification by designing AUCCC, a threshold-free and robust metric, and proposed Uncertainty Distillation to boost performance, showing consistent improvements over baselines on image datasets.
Emergence of artificial intelligence techniques in biomedical applications urges the researchers to pay more attention on the uncertainty quantification (UQ) in machine-assisted medical decision making. For classification tasks, prior studies on UQ are difficult to compare with each other, due to the lack of a unified quantitative evaluation metric. Considering that well-performing UQ models ought to know when the classification models act incorrectly, we design a new evaluation metric, area under Confidence-Classification Characteristic curves (AUCCC), to quantitatively evaluate the performance of the UQ models. AUCCC is threshold-free, robust to perturbation, and insensitive to the classification performance. We evaluate several UQ methods (e.g., max softmax output) with AUCCC to validate its effectiveness. Furthermore, a simple scheme, named Uncertainty Distillation (UDist), is developed to boost the UQ performance, where a confidence model is distilling the confidence estimated by deep ensembles. The proposed method is easy to implement; it consistently outperforms strong baselines on natural and medical image datasets in our experiments.