Achieving Risk Control in Online Learning Settings
This work addresses the need for reliable risk control in online learning settings, which is crucial for applications like time-series analysis and computer vision, though it is incremental as it extends conformal prediction to a broader class of problems.
The paper tackles the problem of providing rigorous uncertainty quantification for online learning models by developing a framework that constructs uncertainty sets to provably control risks like coverage or F1 scores, even under drastic distribution shifts, and demonstrates its utility on real-world datasets and an image-depth estimation problem.
To provide rigorous uncertainty quantification for online learning models, we develop a framework for constructing uncertainty sets that provably control risk -- such as coverage of confidence intervals, false negative rate, or F1 score -- in the online setting. This extends conformal prediction to apply to a larger class of online learning problems. Our method guarantees risk control at any user-specified level even when the underlying data distribution shifts drastically, even adversarially, over time in an unknown fashion. The technique we propose is highly flexible as it can be applied with any base online learning algorithm (e.g., a deep neural network trained online), requiring minimal implementation effort and essentially zero additional computational cost. We further extend our approach to control multiple risks simultaneously, so the prediction sets we generate are valid for all given risks. To demonstrate the utility of our method, we conduct experiments on real-world tabular time-series data sets showing that the proposed method rigorously controls various natural risks. Furthermore, we show how to construct valid intervals for an online image-depth estimation problem that previous sequential calibration schemes cannot handle.