On Training-Conditional Conformal Prediction and Binomial Proportion Confidence Intervals
This work is significant for the control systems community, particularly for those relying on statistical safety certification of dynamical systems, as it highlights the limitations of Conformal Prediction in this context.
The authors tackled the problem of estimating the expectation of a Bernoulli random variable and found that training-conditional Conformal Prediction does not provide valid safety guarantees, with traditional Binomial Proportion Confidence Intervals being more suitable. This result affects the statistical safety certification of dynamical systems.
Estimating the expectation of a Bernoulli random variable based on N independent trials is a classical problem in statistics, typically addressed using Binomial Proportion Confidence Intervals (BPCI). In the control systems community, many critical tasks-such as certifying the statistical safety of dynamical systems-can be formulated as BPCI problems. Conformal Prediction (CP), a distribution-free technique for uncertainty quantification, has gained significant attention in recent years and has been applied to various control systems problems, particularly to address uncertainties in learned dynamics or controllers. A variant known as training-conditional CP was recently employed to tackle the problem of safety certification. In this note, we highlight that the use of training-conditional CP in this context does not provide valid safety guarantees. We demonstrate why CP is unsuitable for BPCI problems and argue that traditional BPCI methods are better suited for statistical safety certification.