On Calibration and Out-of-domain Generalization
This work addresses the problem of out-of-domain generalization for machine learning practitioners, offering a practical method to enhance model robustness, though it builds on existing invariance techniques.
The paper tackles the challenge of out-of-domain generalization by linking it to model calibration, showing that multi-domain calibration can serve as a trainable surrogate to improve performance on unseen domains, with significant gains demonstrated on datasets like WILDS and Colored MNIST.
Out-of-domain (OOD) generalization is a significant challenge for machine learning models. Many techniques have been proposed to overcome this challenge, often focused on learning models with certain invariance properties. In this work, we draw a link between OOD performance and model calibration, arguing that calibration across multiple domains can be viewed as a special case of an invariant representation leading to better OOD generalization. Specifically, we show that under certain conditions, models which achieve \emph{multi-domain calibration} are provably free of spurious correlations. This leads us to propose multi-domain calibration as a measurable and trainable surrogate for the OOD performance of a classifier. We therefore introduce methods that are easy to apply and allow practitioners to improve multi-domain calibration by training or modifying an existing model, leading to better performance on unseen domains. Using four datasets from the recently proposed WILDS OOD benchmark, as well as the Colored MNIST dataset, we demonstrate that training or tuning models so they are calibrated across multiple domains leads to significantly improved performance on unseen test domains. We believe this intriguing connection between calibration and OOD generalization is promising from both a practical and theoretical point of view.