Distribution-free calibration guarantees for histogram binning without sample splitting
This work addresses the need for reliable calibration methods in machine learning, particularly for applications like credit default prediction, by offering a more data-efficient approach without sample splitting, though it is incremental as it builds on existing histogram binning techniques.
The paper tackles the problem of providing theoretical calibration guarantees for histogram binning without requiring sample splitting, which avoids data inefficiency, and proves such guarantees using a Markov property of order statistics, with practical recommendations for bin selection and a new tool called validity plots for calibration assessment.
We prove calibration guarantees for the popular histogram binning (also called uniform-mass binning) method of Zadrozny and Elkan [2001]. Histogram binning has displayed strong practical performance, but theoretical guarantees have only been shown for sample split versions that avoid 'double dipping' the data. We demonstrate that the statistical cost of sample splitting is practically significant on a credit default dataset. We then prove calibration guarantees for the original method that double dips the data, using a certain Markov property of order statistics. Based on our results, we make practical recommendations for choosing the number of bins in histogram binning. In our illustrative simulations, we propose a new tool for assessing calibration -- validity plots -- which provide more information than an ECE estimate. Code for this work will be made publicly available at https://github.com/aigen/df-posthoc-calibration.