Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples
This addresses the challenge of maintaining reliable uncertainty estimates in image classifiers under distribution shift without requiring labeled calibration data, though it is incremental as it builds on existing conformal prediction frameworks.
The paper tackles the problem of recalibrating conformal predictors for new data distributions using only unlabeled examples, as labeled data is often unavailable, and proposes a method that provides excellent uncertainty estimates under natural distribution shifts with provable guarantees for a specific shift model.
Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes containing the correct class with a user-specified probability based on the classifier's probability estimates. To provide such sets, conformal predictors often estimate a cutoff threshold for the probability estimates based on a calibration set. Conformal predictors guarantee reliability only when the calibration set is from the same distribution as the test set. Therefore, conformal predictors need to be recalibrated for new distributions. However, in practice, labeled data from new distributions is rarely available, making calibration infeasible. In this work, we consider the problem of predicting the cutoff threshold for a new distribution based on unlabeled examples. While it is impossible in general to guarantee reliability when calibrating based on unlabeled examples, we propose a method that provides excellent uncertainty estimates under natural distribution shifts, and provably works for a specific model of a distribution shift.