Time-uniform conformal and PAC prediction
This work addresses the need for reliable uncertainty quantification in high-stakes decision-making with streaming data, providing a novel sequential extension that is anytime-valid, though it builds incrementally on existing conformal prediction frameworks.
The authors tackled the problem of uncertainty quantification in sequential settings where traditional conformal prediction methods fail without a fixed sample size and cannot handle updated predictions, by developing an extension to conformal and PAC prediction frameworks that ensures expected coverage at any time, even with data-dependent choices, and demonstrated validity on simulated and real datasets.
Given that machine learning algorithms are increasingly being deployed to aid in high stakes decision-making, uncertainty quantification methods that wrap around these black box models such as conformal prediction have received much attention in recent years. In sequential settings, where data are observed/generated in a streaming fashion, traditional conformal methods do not provide any guarantee without fixing the sample size. More importantly, traditional conformal methods cannot cope with sequentially updated predictions. As such, we develop an extension of the conformal prediction and related probably approximately correct (PAC) prediction frameworks to sequential settings where the number of data points is not fixed in advance. The resulting prediction sets are anytime-valid in that their expected coverage is at the required level at any time chosen by the analyst even if this choice depends on the data. We present theoretical guarantees for our proposed methods and demonstrate their validity and utility on simulated and real datasets.