Conformal Risk Control
This work provides a general framework for risk control in machine learning, applicable across domains like computer vision and NLP, though it is incremental as it builds on existing conformal prediction methods.
The authors tackled the problem of extending conformal prediction to control the expected value of any monotone loss function, resulting in a tight algorithm with an O(1/n) factor guarantee and demonstrated applications in computer vision and NLP to bound metrics like false negative rate and F1-score.
We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score.