An Investigation of how Label Smoothing Affects Generalization
This work addresses a foundational problem for machine learning researchers and practitioners by providing theoretical insights into label smoothing, though it is incremental as it builds on existing empirical evidence.
The paper tackles the lack of mathematical understanding of why label smoothing improves generalization by proposing a theoretical framework that shows its benefits in controlling generalization loss, particularly in label noise settings, and predicts an optimal label smoothing point, with experiments confirming these predictions.
It has been hypothesized that label smoothing can reduce overfitting and improve generalization, and current empirical evidence seems to corroborate these effects. However, there is a lack of mathematical understanding of when and why such empirical improvements occur. In this paper, as a step towards understanding why label smoothing is effective, we propose a theoretical framework to show how label smoothing provides in controlling the generalization loss. In particular, we show that this benefit can be precisely formulated and identified in the label noise setting, where the training is partially mislabeled. Our theory also predicts the existence of an optimal label smoothing point, a single value for the label smoothing hyperparameter that minimizes generalization loss. Extensive experiments are done to confirm the predictions of our theory. We believe that our findings will help both theoreticians and practitioners understand label smoothing, and better apply them to real-world datasets.