Calibrated Top-1 Uncertainty estimates for classification by score based models
This addresses the issue of unreliable uncertainty estimates in classification models, which is crucial for applications requiring trustworthy predictions, though it is incremental as it builds on existing methods.
The paper tackles the problem of poor calibration in deep learning uncertainty estimates, achieving less than 1% calibration error by focusing on Top-1 error probability, which significantly improves scores over benchmarks.
While the accuracy of modern deep learning models has significantly improved in recent years, the ability of these models to generate uncertainty estimates has not progressed to the same degree. Uncertainty methods are designed to provide an estimate of class probabilities when predicting class assignment. While there are a number of proposed methods for estimating uncertainty, they all suffer from a lack of calibration: predicted probabilities can be off from empirical ones by a few percent or more. By restricting the scope of our predictions to only the probability of Top-1 error, we can decrease the calibration error of existing methods to less than one percent. As a result, the scores of the methods also improve significantly over benchmarks.