Label Smoothing++: Enhanced Label Regularization for Training Neural Networks
This work addresses overconfidence and overfitting issues in neural network training, offering an incremental improvement over standard label smoothing by better preserving inter-class relationships.
The paper tackles the problem of overconfidence and overfitting in neural networks by proposing Label Smoothing++, a label regularization strategy that assigns non-zero probabilities to non-target classes while accounting for inter-class relationships, resulting in improved generalization capabilities as demonstrated through experiments on multiple datasets.
Training neural networks with one-hot target labels often results in overconfidence and overfitting. Label smoothing addresses this issue by perturbing the one-hot target labels by adding a uniform probability vector to create a regularized label. Although label smoothing improves the network's generalization ability, it assigns equal importance to all the non-target classes, which destroys the inter-class relationships. In this paper, we propose a novel label regularization training strategy called Label Smoothing++, which assigns non-zero probabilities to non-target classes and accounts for their inter-class relationships. Our approach uses a fixed label for the target class while enabling the network to learn the labels associated with non-target classes. Through extensive experiments on multiple datasets, we demonstrate how Label Smoothing++ mitigates overconfident predictions while promoting inter-class relationships and generalization capabilities.