Confidence Calibration with Bounded Error Using Transformations
This addresses the need for accurate uncertainty estimation in safety-critical applications such as autonomous vehicles, though it appears incremental as it builds on existing calibration methods.
The paper tackles the problem of improving confidence calibration for neural networks, especially in high-dimensional settings like ImageNet, by introducing Hoki, a method that applies random transformations to logits and achieves better performance than state-of-the-art calibration algorithms.
As machine learning techniques become widely adopted in new domains, especially in safety-critical systems such as autonomous vehicles, it is crucial to provide accurate output uncertainty estimation. As a result, many approaches have been proposed to calibrate neural networks to accurately estimate the likelihood of misclassification. However, while these methods achieve low calibration error, there is space for further improvement, especially in large-dimensional settings such as ImageNet. In this paper, we introduce a calibration algorithm, named Hoki, that works by applying random transformations to the neural network logits. We provide a sufficient condition for perfect calibration based on the number of label prediction changes observed after applying the transformations. We perform experiments on multiple datasets and show that the proposed approach generally outperforms state-of-the-art calibration algorithms across multiple datasets and models, especially on the challenging ImageNet dataset. Finally, Hoki is scalable as well, as it requires comparable execution time to that of temperature scaling.