Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction
This work addresses the problem of reliable prediction for a wide range of applications, particularly those requiring robust calibration across subpopulations.
The authors tackled the problem of model calibration, achieving finite-sample calibration for a broad class of prediction problems, with their method transforming any perfectly in-sample calibrated predictor into a set-valued predictor that outputs at least one marginally calibrated point prediction. Their framework yields novel prediction intervals with quantile-conditional coverage.
Ensuring model calibration is critical for reliable prediction, yet popular distribution-free methods such as histogram binning and isotonic regression offer only asymptotic guarantees. We introduce a unified framework for Venn and Venn-Abers calibration that extends Vovk's approach beyond binary classification to a broad class of prediction problems defined by generic loss functions. Our method transforms any perfectly in-sample calibrated predictor into a set-valued predictor that, in finite samples, outputs at least one marginally calibrated point prediction. These set predictions shrink asymptotically and converge to a single conditionally calibrated prediction, capturing epistemic uncertainty. We further propose Venn multicalibration, a new approach for achieving finite-sample calibration across subpopulations. For quantile loss, our framework recovers group-conditional and multicalibrated conformal prediction as special cases and yields novel prediction intervals with quantile-conditional coverage.