Rethinking Early Stopping: Refine, Then Calibrate
This addresses the issue of accurate and interpretable probabilistic predictions for machine learning practitioners, though it is incremental as it builds on existing calibration techniques.
The paper tackles the problem that training classifiers to minimize validation loss leads to suboptimal probabilistic predictions, as calibration and refinement errors are not minimized simultaneously. The proposed method of first minimizing refinement error during training and then calibrating post-hoc consistently improves performance across diverse classification tasks.
Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses, such as cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we present a novel variational formulation of the calibration-refinement decomposition that sheds new light on post-hoc calibration, and enables rapid estimation of the different terms. Equipped with this new perspective, we provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training. Selecting the best epoch based on validation loss thus leads to a compromise point that is suboptimal for both terms. To address this, we propose minimizing refinement error only during training (Refine,...), before minimizing calibration error post hoc, using standard techniques (...then Calibrate). Our method integrates seamlessly with any classifier and consistently improves performance across diverse classification tasks.