A Variational Estimator for $L_p$ Calibration Errors
This work addresses the problem of reliable probability calibration for machine learning practitioners, offering an incremental improvement by extending existing variational methods to new divergence types.
The paper tackles the challenge of accurately estimating calibration errors in machine learning, particularly for multiclass settings, by extending a variational framework to cover a broad class of $L_p$ divergence-induced errors, resulting in a method that avoids overestimation and separates over- and under-confidence.
Calibration$\unicode{x2014}$the problem of ensuring that predicted probabilities align with observed class frequencies$\unicode{x2014}$is a basic desideratum for reliable prediction with machine learning systems. Calibration error is traditionally assessed via a divergence function, using the expected divergence between predictions and empirical frequencies. Accurately estimating this quantity is challenging, especially in the multiclass setting. Here, we show how to extend a recent variational framework for estimating calibration errors beyond divergences induced induced by proper losses, to cover a broad class of calibration errors induced by $L_p$ divergences. Our method can separate over- and under-confidence and, unlike non-variational approaches, avoids overestimation. We provide extensive experiments and integrate our code in the open-source package probmetrics (https://github.com/dholzmueller/probmetrics) for evaluating calibration errors.