Calibrating sufficiently
This work addresses calibration issues for machine learning practitioners, but it appears incremental as it builds on existing concepts like sufficiency and probing reduction.
The paper tackles the problem of grouping loss in probabilistic classifier calibration, showing that the probing reduction approach produces an estimator that reduces this loss.
When probabilistic classifiers are trained and calibrated, the so-called grouping loss component of the calibration loss can easily be overlooked. Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise. We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency. We revisit the probing reduction approach of Langford & Zadrozny (2005) and find that it produces an estimator of probabilistic classifiers that reduces grouping loss. Finally, we discuss Brier curves as tools to support training and 'sufficient' calibration of probabilistic classifiers.