Spline-Based Probability Calibration
This addresses the issue of overconfident scores in deep learning and other domains, offering a practical solution for improving probability calibration.
The paper tackles the problem of outputting well-calibrated probabilities in classification by proposing SplineCalib, a robust, non-parametric method using smoothing splines, which improves log-loss and accuracy on various problems.
In many classification problems it is desirable to output well-calibrated probabilities on the different classes. We propose a robust, non-parametric method of calibrating probabilities called SplineCalib that utilizes smoothing splines to determine a calibration function. We demonstrate how applying certain transformations as part of the calibration process can improve performance on problems in deep learning and other domains where the scores tend to be "overconfident". We adapt the approach to multi-class problems and find that better calibration can improve accuracy as well as log-loss by better resolving uncertain cases. Finally, we present a cross-validated approach to calibration which conserves data. Significant improvements to log-loss and accuracy are shown on several different problems. We also introduce the ml-insights python package which contains an implementation of the SplineCalib algorithm.