Rethinking Soft Label in Label Distribution Learning Perspective
This work addresses model calibration issues for image classification tasks, offering improvements in both calibration and accuracy, though it appears incremental as it builds on existing LDL and data augmentation techniques.
The paper tackled the problem of model overconfidence and poor calibration in CNNs by hypothesizing that a gap in supervision criteria leads to overconfidence, and it investigated label distribution learning (LDL) to enhance calibration. The result showed that training with LDL and data augmentation achieved lower expected calibration error (ECE) and higher generalization performance on datasets like CIFAR10, CIFAR100, STL10, and ImageNet.
The primary goal of training in early convolutional neural networks (CNN) is the higher generalization performance of the model. However, as the expected calibration error (ECE), which quantifies the explanatory power of model inference, was recently introduced, research on training models that can be explained is in progress. We hypothesized that a gap in supervision criteria during training and inference leads to overconfidence, and investigated that performing label distribution learning (LDL) would enhance the model calibration in CNN training. To verify this assumption, we used a simple LDL setting with recent data augmentation techniques. Based on a series of experiments, the following results are obtained: 1) State-of-the-art KD methods significantly impede model calibration. 2) Training using LDL with recent data augmentation can have excellent effects on model calibration and even in generalization performance. 3) Online LDL brings additional improvements in model calibration and accuracy with long training, especially in large-size models. Using the proposed approach, we simultaneously achieved a lower ECE and higher generalization performance for the image classification datasets CIFAR10, 100, STL10, and ImageNet. We performed several visualizations and analyses and witnessed several interesting behaviors in CNN training with the LDL.