Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration
This addresses the issue of unreliable confidence estimates in AI predictions for real-world applications where test data distributions differ from training, though it is an incremental improvement over existing calibration methods.
The paper tackles the problem of deep neural networks being poorly calibrated for out-of-distribution test data by proposing a meta-set-based cascaded temperature regression method that tailors scaling functions to distinct test sets, achieving effective calibration across datasets like MNIST, CIFAR-10, and TinyImageNet.
Although deep neural networks yield high classification accuracy given sufficient training data, their predictions are typically overconfident or under-confident, i.e., the prediction confidences cannot truly reflect the accuracy. Post-hoc calibration tackles this problem by calibrating the prediction confidences without re-training the classification model. However, current approaches assume congruence between test and validation data distributions, limiting their applicability to out-of-distribution scenarios. To this end, we propose a novel meta-set-based cascaded temperature regression method for post-hoc calibration. Our method tailors fine-grained scaling functions to distinct test sets by simulating various domain shifts through data augmentation on the validation set. We partition each meta-set into subgroups based on predicted category and confidence level, capturing diverse uncertainties. A regression network is then trained to derive category-specific and confidence-level-specific scaling, achieving calibration across meta-sets. Extensive experimental results on MNIST, CIFAR-10, and TinyImageNet demonstrate the effectiveness of the proposed method.