Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach
This addresses the challenge of reliable uncertainty estimation in machine learning models when faced with domain shifts, which is crucial for applications like autonomous driving or medical diagnosis, though it is incremental as it builds on existing distributionally robust learning methods.
The paper tackles the problem of learning calibrated uncertainties under domain shifts, where training and test distributions differ, by proposing a distributionally robust learning framework that uses a differentiable density ratio estimator to adjust prediction uncertainties, resulting in significant improvements in cross-domain performance for tasks like unsupervised domain adaptation and semi-supervised learning.
We propose a framework for learning calibrated uncertainties under domain shifts, where the source (training) distribution differs from the target (test) distribution. We detect such domain shifts via a differentiable density ratio estimator and train it together with the task network, composing an adjusted softmax predictive form concerning domain shift. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We employ it to adjust the uncertainty of prediction in the task network. This idea of using the density ratio is based on the distributionally robust learning (DRL) framework, which accounts for the domain shift by adversarial risk minimization. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks, such as unsupervised domain adaptation (UDA) and semi-supervised learning (SSL). On these tasks, methods like self-training and FixMatch use uncertainties to select confident pseudo-labels for re-training. Our experiments show that the introduction of DRL leads to significant improvements in cross-domain performance. We also show that the estimated density ratios align with human selection frequencies, suggesting a positive correlation with a proxy of human perceived uncertainties.