Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift
This work addresses the issue of unreliable uncertainty quantification in machine learning under distribution shift, which is critical for real-world applications, but it is incremental as it builds on existing conformal prediction and domain adaptation techniques.
The paper tackles the problem of conformal prediction's coverage guarantees failing under distribution shift by analyzing pseudo-calibration under a bounded label-conditional covariate shift model, resulting in a method that mitigates coverage degradation while maintaining nontrivial prediction set sizes in numerical experiments.
Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.