Online Selective Conformal Prediction: Errors and Solutions
This addresses reliability issues in sequential prediction intervals for researchers and practitioners in machine learning, though it is incremental as it builds on existing conformal prediction methods.
The paper tackled errors in existing calibration selection strategies for online selective conformal prediction, which can break exchangeability and compromise coverage guarantees, and proposed novel strategies that provably ensure selection-conditional coverage and false coverage rate control, supported by experimental evidence on tradeoffs.
In online selective conformal inference, data arrives sequentially, and prediction intervals are constructed only when an online selection rule is met. Since online selections may break the exchangeability between the selected test datum and the rest of the data, one must correct for this by suitably selecting the calibration data. In this paper, we evaluate existing calibration selection strategies and pinpoint some fundamental errors in the associated claims that guarantee selection-conditional coverage and control of the false coverage rate (FCR). To address these shortcomings, we propose novel calibration selection strategies that provably preserve the exchangeability of the calibration data and the selected test datum. Consequently, we demonstrate that online selective conformal inference with these strategies guarantees both selection-conditional coverage and FCR control. Our theoretical findings are supported by experimental evidence examining tradeoffs between valid methods.