A Unifying Theory of Distance from Calibration
This work addresses a fundamental issue in machine learning for researchers and practitioners by providing a rigorous framework to evaluate and compare calibration measures, establishing theoretical bounds and justifying practical metric choices.
The paper tackles the problem of quantifying distance from calibration for probabilistic predictors by proposing a ground-truth notion based on the ℓ1 distance to the nearest perfectly calibrated predictor, and identifies three consistent calibration measures (smooth, interval, and Laplace kernel calibration) with quadratic approximations shown to be information-theoretically optimal in a prediction-only access model.
We study the fundamental question of how to define and measure the distance from calibration for probabilistic predictors. While the notion of perfect calibration is well-understood, there is no consensus on how to quantify the distance from perfect calibration. Numerous calibration measures have been proposed in the literature, but it is unclear how they compare to each other, and many popular measures such as Expected Calibration Error (ECE) fail to satisfy basic properties like continuity. We present a rigorous framework for analyzing calibration measures, inspired by the literature on property testing. We propose a ground-truth notion of distance from calibration: the $\ell_1$ distance to the nearest perfectly calibrated predictor. We define a consistent calibration measure as one that is polynomially related to this distance. Applying our framework, we identify three calibration measures that are consistent and can be estimated efficiently: smooth calibration, interval calibration, and Laplace kernel calibration. The former two give quadratic approximations to the ground truth distance, which we show is information-theoretically optimal in a natural model for measuring calibration which we term the prediction-only access model. Our work thus establishes fundamental lower and upper bounds on measuring the distance to calibration, and also provides theoretical justification for preferring certain metrics (like Laplace kernel calibration) in practice.