Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
This work addresses the need for reliable risk control in predictive algorithms across various domains, offering a general solution that is not incremental but introduces a novel paradigm.
The paper tackles the problem of calibrating machine learning models to achieve explicit, finite-sample statistical guarantees, such as false discovery rate control in multi-label classification, without requiring model refitting. It introduces a framework that reframes risk control as multiple hypothesis testing, providing new calibration methods validated on tasks like computer vision and medical data.
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use the framework to provide new calibration methods for several core machine learning tasks, with detailed worked examples in computer vision and tabular medical data.