Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
This addresses the need for reliable probability estimates in classification tasks, particularly for deep learning applications, though it is incremental as it builds on existing post-hoc calibration techniques.
The paper tackles the problem of uncalibrated posterior probabilities in classifiers, especially deep neural networks, by introducing Meta-Cal, a post-hoc calibration method that incorporates constraints beyond low calibration errors, resulting in significant outperformance over state-of-the-art methods on datasets like CIFAR-10, CIFAR-100, and ImageNet.
In many applications, it is desirable that a classifier not only makes accurate predictions, but also outputs calibrated posterior probabilities. However, many existing classifiers, especially deep neural network classifiers, tend to be uncalibrated. Post-hoc calibration is a technique to recalibrate a model by learning a calibration map. Existing approaches mostly focus on constructing calibration maps with low calibration errors, however, this quality is inadequate for a calibrator being useful. In this paper, we introduce two constraints that are worth consideration in designing a calibration map for post-hoc calibration. Then we present Meta-Cal, which is built from a base calibrator and a ranking model. Under some mild assumptions, two high-probability bounds are given with respect to these constraints. Empirical results on CIFAR-10, CIFAR-100 and ImageNet and a range of popular network architectures show our proposed method significantly outperforms the current state of the art for post-hoc multi-class classification calibration.