A Confidence Interval for the $\ell_2$ Expected Calibration Error
This work addresses the need for statistical evaluation of calibration in machine learning models, which is crucial for reliable probabilistic predictions, though it is incremental in nature.
The paper tackles the problem of rigorously evaluating model calibration by developing confidence intervals for the ℓ₂ Expected Calibration Error (ECE), showing that their methods produce valid intervals with shorter lengths compared to resampling-based approaches.
Recent advances in machine learning have significantly improved prediction accuracy in various applications. However, ensuring the calibration of probabilistic predictions remains a significant challenge. Despite efforts to enhance model calibration, the rigorous statistical evaluation of model calibration remains less explored. In this work, we develop confidence intervals the $\ell_2$ Expected Calibration Error (ECE). We consider top-1-to-$k$ calibration, which includes both the popular notion of confidence calibration as well as full calibration. For a debiased estimator of the ECE, we show asymptotic normality, but with different convergence rates and asymptotic variances for calibrated and miscalibrated models. We develop methods to construct asymptotically valid confidence intervals for the ECE, accounting for this behavior as well as non-negativity. Our theoretical findings are supported through extensive experiments, showing that our methods produce valid confidence intervals with shorter lengths compared to those obtained by resampling-based methods.