Robust Calibration with Multi-domain Temperature Scaling
This addresses the challenge of reliable model deployment in high-stakes applications where distribution shifts are common, representing an incremental improvement over existing calibration techniques.
The paper tackles the problem of uncertainty quantification under distribution shifts by proposing a multi-domain temperature scaling method, which improves calibration robustness and outperforms existing methods on both in-distribution and out-of-distribution test sets across three benchmark datasets.
Uncertainty quantification is essential for the reliable deployment of machine learning models to high-stakes application domains. Uncertainty quantification is all the more challenging when training distribution and test distribution are different, even the distribution shifts are mild. Despite the ubiquity of distribution shifts in real-world applications, existing uncertainty quantification approaches mainly study the in-distribution setting where the train and test distributions are the same. In this paper, we develop a systematic calibration model to handle distribution shifts by leveraging data from multiple domains. Our proposed method -- multi-domain temperature scaling -- uses the heterogeneity in the domains to improve calibration robustness under distribution shift. Through experiments on three benchmark data sets, we find our proposed method outperforms existing methods as measured on both in-distribution and out-of-distribution test sets.