Mirror Online Conformal Prediction with Intermittent Feedback
This addresses the challenge of maintaining calibration guarantees in online settings with sporadic feedback, which is incremental but improves over prior methods that sacrificed coverage for regret guarantees.
The paper tackles the problem of runtime calibration for AI models with intermittent feedback by introducing IM-OCP, a framework that guarantees long-term coverage and sub-linear regret deterministically and in expectation.
Online conformal prediction enables the runtime calibration of a pre-trained artificial intelligence model using feedback on its performance. Calibration is achieved through set predictions that are updated via online rules so as to ensure long-term coverage guarantees. While recent research has demonstrated the benefits of incorporating prior knowledge into the calibration process, this has come at the cost of replacing coverage guarantees with less tangible regret guarantees based on the quantile loss. This work introduces intermittent mirror online conformal prediction (IM-OCP), a novel runtime calibration framework that integrates prior knowledge, operates under potentially intermittent feedback, and features minimal memory complexity. IM-OCP guarantees long-term coverage and sub-linear regret, both of which hold deterministically for any given data sequence and in expectation with respect to the intermittent feedback.