Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness
This addresses calibration issues in machine learning classifiers, particularly for image classification, but is incremental as it builds on existing methods like focal loss and temperature scaling.
The paper explains why focal loss training often yields better-calibrated classifiers than cross-entropy by decomposing it into a confidence-raising transformation and a proper loss, and introduces focal temperature scaling, which outperforms standard temperature scaling on three image classification datasets.
Proper losses such as cross-entropy incentivize classifiers to produce class probabilities that are well-calibrated on the training data. Due to the generalization gap, these classifiers tend to become overconfident on the test data, mandating calibration methods such as temperature scaling. The focal loss is not proper, but training with it has been shown to often result in classifiers that are better calibrated on test data. Our first contribution is a simple explanation about why focal loss training often leads to better calibration than cross-entropy training. For this, we prove that focal loss can be decomposed into a confidence-raising transformation and a proper loss. This is why focal loss pushes the model to provide under-confident predictions on the training data, resulting in being better calibrated on the test data, due to the generalization gap. Secondly, we reveal a strong connection between temperature scaling and focal loss through its confidence-raising transformation, which we refer to as the focal calibration map. Thirdly, we propose focal temperature scaling - a new post-hoc calibration method combining focal calibration and temperature scaling. Our experiments on three image classification datasets demonstrate that focal temperature scaling outperforms standard temperature scaling.