Distribution-informed Online Conformal Prediction
This work addresses uncertainty quantification for machine learning systems in dynamic environments, offering a practical improvement over existing methods.
The paper tackles the problem of overly conservative prediction sets in online conformal prediction under data distribution shifts by proposing Conformal Optimistic Prediction (COP), which incorporates data patterns to produce tighter prediction sets while maintaining valid coverage guarantees, achieving shorter prediction intervals than baselines in experiments.
Conformal prediction provides a pivotal and flexible technique for uncertainty quantification by constructing prediction sets with a predefined coverage rate. Many online conformal prediction methods have been developed to address data distribution shifts in fully adversarial environments, resulting in overly conservative prediction sets. We propose Conformal Optimistic Prediction (COP), an online conformal prediction algorithm incorporating underlying data pattern into the update rule. Through estimated cumulative distribution function of non-conformity scores, COP produces tighter prediction sets when predictable pattern exists, while retaining valid coverage guarantees even when estimates are inaccurate. We establish a joint bound on coverage and regret, which further confirms the validity of our approach. We also prove that COP achieves distribution-free, finite-sample coverage under arbitrary learning rates and can converge when scores are $i.i.d.$. The experimental results also show that COP can achieve valid coverage and construct shorter prediction intervals than other baselines.