Conformal Predictive Systems Under Covariate Shift
This work addresses the problem of making predictive systems robust to covariate shifts for practitioners in machine learning and statistics, representing an incremental extension of existing methods.
The paper tackles the limitation of Conformal Predictive Systems (CPS) to IID scenarios by extending them to handle covariate shifts, proposing Weighted CPS (WCPS) that uses likelihood ratios to construct calibrated predictive distributions, with empirical results showing probabilistic calibration under covariate shift.
Conformal Predictive Systems (CPS) offer a versatile framework for constructing predictive distributions, allowing for calibrated inference and informative decision-making. However, their applicability has been limited to scenarios adhering to the Independent and Identically Distributed (IID) model assumption. This paper extends CPS to accommodate scenarios characterized by covariate shifts. We therefore propose Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions. This extension enables the construction of nonparametric predictive distributions capable of handling covariate shifts. We present theoretical underpinnings and conjectures regarding the validity and efficacy of WCPS and demonstrate its utility through empirical evaluations on both synthetic and real-world datasets. Our simulation experiments indicate that WCPS are probabilistically calibrated under covariate shift.