Revisiting Confidence Estimation: Towards Reliable Failure Prediction
This addresses a fundamental issue in risk-sensitive applications where reliable confidence estimation is crucial for trust in AI predictions, though it is incremental as it builds on existing calibration and OOD detection methods.
The paper tackles the problem that many confidence estimation methods worsen the separation between correct and misclassified predictions, making failure prediction unreliable, and proposes a method to enlarge the confidence gap by finding flat minima, achieving state-of-the-art performance in various classification scenarios.
Reliable confidence estimation is a challenging yet fundamental requirement in many risk-sensitive applications. However, modern deep neural networks are often overconfident for their incorrect predictions, i.e., misclassified samples from known classes, and out-of-distribution (OOD) samples from unknown classes. In recent years, many confidence calibration and OOD detection methods have been developed. In this paper, we find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We investigate this problem and reveal that popular calibration and OOD detection methods often lead to worse confidence separation between correctly classified and misclassified examples, making it difficult to decide whether to trust a prediction or not. Finally, we propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance under various settings including balanced, long-tailed, and covariate-shift classification scenarios. Our study not only provides a strong baseline for reliable confidence estimation but also acts as a bridge between understanding calibration, OOD detection, and failure prediction. The code is available at \url{https://github.com/Impression2805/FMFP}.