Conformal Prediction Under Distribution Shift: A COVID-19 Natural Experiment
This work addresses the reliability of conformal prediction under distribution shift for practitioners in fields like supply chain management, offering a decision framework for monitoring and retraining, though it is incremental as it builds on existing methods with new empirical insights.
The paper tackled the problem of conformal prediction guarantees degrading under distribution shift by using COVID-19 as a natural experiment across supply chain tasks, finding that coverage drops varied widely from 0% to 86.7% and that catastrophic failures correlated with single-feature dependence, with quarterly retraining restoring coverage from 22% to 41% for vulnerable tasks.
Conformal prediction guarantees degrade under distribution shift. We study this using COVID-19 as a natural experiment across 8 supply chain tasks. Despite identical severe feature turnover (Jaccard approximately 0), coverage drops vary from 0% to 86.7%, spanning two orders of magnitude. Using SHapley Additive exPlanations (SHAP) analysis, we find catastrophic failures correlate with single-feature dependence (rho = 0.714, p = 0.047). Catastrophic tasks concentrate importance in one feature (4.5x increase), while robust tasks redistribute across many (10-20x). Quarterly retraining restores catastrophic task coverage from 22% to 41% (+19 pp, p = 0.04), but provides no benefit for robust tasks (99.8% coverage). Exploratory analysis of 4 additional tasks with moderate feature stability (Jaccard 0.13-0.86) reveals feature stability, not concentration, determines robustness, suggesting concentration effects apply specifically to severe shifts. We provide a decision framework: monitor SHAP concentration before deployment; retrain quarterly if vulnerable (>40% concentration); skip retraining if robust.