Plots of the cumulative differences between observed and expected values of ordered Bernoulli variates
This provides a more straightforward visualization tool for researchers and practitioners assessing calibration in probabilistic models, though it is incremental as it builds on existing reliability diagrams.
The paper tackles the problem of visualizing miscalibration in probabilistic predictions by introducing plots of cumulative differences between observed and expected values, which avoid the need for arbitrary binning or kernel width selection and display miscalibration directly as slopes of secant lines.
Many predictions are probabilistic in nature; for example, a prediction could be for precipitation tomorrow, but with only a 30 percent chance. Given both the predictions and the actual outcomes, "reliability diagrams" (also known as "calibration plots") help detect and diagnose statistically significant discrepancies between the predictions and the outcomes. The canonical reliability diagrams are based on histogramming the observed and expected values of the predictions; several variants of the standard reliability diagrams propose to replace the hard histogram binning with soft kernel density estimation using smooth convolutional kernels of widths similar to the widths of the bins. In all cases, an important question naturally arises: which widths are best (or are multiple plots with different widths better)? Rather than answering this question, plots of the cumulative differences between the observed and expected values largely avoid the question, by displaying miscalibration directly as the slopes of secant lines for the graphs. Slope is easy to perceive with quantitative precision even when the constant offsets of the secant lines are irrelevant. There is no need to bin or perform kernel density estimation with a somewhat arbitrary kernel.