Transferable Calibration with Lower Bias and Variance in Domain Adaptation
This addresses the need for reliable uncertainty estimation in safety-critical decision-making for domain adaptation applications, representing a novel but incremental improvement in calibration methods.
The paper tackles the problem of predictive uncertainty calibration in Domain Adaptation, where models often sacrifice calibration for accuracy due to domain shift and lack of target labels, and proposes Transferable Calibration (TransCal) to achieve more accurate calibration with lower bias and variance, showing efficacy theoretically and empirically.
Domain Adaptation (DA) enables transferring a learning machine from a labeled source domain to an unlabeled target one. While remarkable advances have been made, most of the existing DA methods focus on improving the target accuracy at inference. How to estimate the predictive uncertainty of DA models is vital for decision-making in safety-critical scenarios but remains the boundary to explore. In this paper, we delve into the open problem of Calibration in DA, which is extremely challenging due to the coexistence of domain shift and the lack of target labels. We first reveal the dilemma that DA models learn higher accuracy at the expense of well-calibrated probabilities. Driven by this finding, we propose Transferable Calibration (TransCal) to achieve more accurate calibration with lower bias and variance in a unified hyperparameter-free optimization framework. As a general post-hoc calibration method, TransCal can be easily applied to recalibrate existing DA methods. Its efficacy has been justified both theoretically and empirically.