Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data
This addresses the challenge of maintaining reliable uncertainty quantification in machine learning under real-world distribution shifts, offering a more robust solution for practitioners.
The paper tackles the problem of distribution shifts in conformal prediction, which violate exchangeability assumptions and reduce coverage guarantees, by proposing a method using optimal transport to estimate and mitigate arbitrary shifts with unlabeled data, achieving improved coverage without prior knowledge of the shift type.
Conformal prediction is a distribution-free uncertainty quantification method that has gained popularity in the machine learning community due to its finite-sample guarantees and ease of use. Its most common variant, dubbed split conformal prediction, is also computationally efficient as it boils down to collecting statistics of the model predictions on some calibration data not yet seen by the model. Nonetheless, these guarantees only hold if the calibration and test data are exchangeable, a condition that is difficult to verify and often violated in practice due to so-called distribution shifts. The literature is rife with methods to mitigate the loss in coverage in this non-exchangeable setting, but these methods require some prior information on the type of distribution shift to be expected at test time. In this work, we study this problem via a new perspective, through the lens of optimal transport, and show that it is possible to estimate the loss in coverage and mitigate arbitrary distribution shifts, offering a principled and broadly applicable solution.