Adaptive Set-Mass Calibration with Conformal Prediction
This work addresses the need for reliable probabilities in high-risk domains, offering a practical and scalable framework with theoretical guarantees, though it is incremental as it builds on existing calibration and conformal prediction methods.
The paper tackled the problem of unreliable probabilities in high-risk applications by proposing a set-based notion of calibration called cumulative mass calibration and a corresponding error measure, CMCE, with a new calibration procedure using conformal prediction. The result showed consistent improvements in CMCE and standard metrics like ECE on multi-class image benchmarks, especially with many classes.
Reliable probabilities are critical in high-risk applications, yet common calibration criteria (confidence, class-wise) are only necessary for full distributional calibration, and post-hoc methods often lack distribution-free guarantees. We propose a set-based notion of calibration, cumulative mass calibration, and a corresponding empirical error measure: the Cumulative Mass Calibration Error (CMCE). We develop a new calibration procedure that starts with conformal prediction to obtain a set of labels that gives the desired coverage. We then instantiate two simple post-hoc calibrators: a mass normalization and a temperature scaling-based rule, tuned to the conformal constraint. On multi-class image benchmarks, especially with a large number of classes, our methods consistently improve CMCE and standard metrics (ECE, cw-ECE, MCE) over baselines, delivering a practical, scalable framework with theoretical guarantees.