Measuring Uncertainty Calibration
This work addresses the challenge of reliably measuring calibration error in machine learning models, offering practical tools for real-world applications, though it is incremental as it builds on existing calibration estimation methods.
The paper tackles the problem of estimating the L1 calibration error of binary classifiers from finite datasets, providing an upper bound for classifiers with bounded variation calibration functions and a method to modify classifiers for efficient error bounding without significantly impacting performance, all with non-asymptotic and distribution-free results.
We make two contributions to the problem of estimating the $L_1$ calibration error of a binary classifier from a finite dataset. First, we provide an upper bound for any classifier where the calibration function has bounded variation. Second, we provide a method of modifying any classifier so that its calibration error can be upper bounded efficiently without significantly impacting classifier performance and without any restrictive assumptions. All our results are non-asymptotic and distribution-free. We conclude by providing advice on how to measure calibration error in practice. Our methods yield practical procedures that can be run on real-world datasets with modest overhead.