Stable Localized Conformal Prediction via Transduction
For practitioners using conformal prediction with limited calibration data, this work addresses the practical issue of unstable prediction set sizes, particularly in localized methods.
The paper formalizes set stability in conformal prediction as the variance of conditional expected set size given calibration data, and proposes Stable Conformal Prediction (StCP) using transfer learning to reduce variability without extra target labels. Experiments show StCP yields more stable prediction sets than standard methods, especially with limited calibration data.
Existing evaluations of conformal prediction, such as prediction efficiency and test-conditional coverage, are defined in expectation over the calibration data. In practice, when only one calibration set of limited size is available, prediction sets often exhibit high variability in size, especially for methods with localization. We formalize this concern as set stability, defined as the variance of the conditional expectation of the set size given the calibration data. To improve stability without requiring additional target-task labels, we propose Stable Conformal Prediction (StCP), a transfer learning approach that utilizes labeled source-task data and unlabeled target data. Theoretically, we characterize the marginal coverage and stability of StCP; empirically, it delivers more stable prediction sets than standard conformal prediction methods, especially for those with localization, when calibration data are limited.