For Better or For Worse? Learning Minimum Variance Features With Label Augmentation
This work provides theoretical and empirical insights into the trade-offs of label augmentation, which is important for researchers and practitioners in deep learning to optimize model robustness and performance.
The paper analyzes how label augmentation methods like label smoothing and Mixup cause models to learn minimum variance features, showing that this leads to improved performance on image classification benchmarks but also increases susceptibility to low variance spurious correlations.
Data augmentation has been pivotal in successfully training deep learning models on classification tasks over the past decade. An important subclass of data augmentation techniques - which includes both label smoothing and Mixup - involves modifying not only the input data but also the input label during model training. In this work, we analyze the role played by the label augmentation aspect of such methods. We first prove that linear models on binary classification data trained with label augmentation learn only the minimum variance features in the data, while standard training (which includes weight decay) can learn higher variance features. We then use our techniques to show that even for nonlinear models and general data distributions, the label smoothing and Mixup losses are lower bounded by a function of the model output variance. Lastly, we demonstrate empirically that this aspect of label smoothing and Mixup can be a positive and a negative. On the one hand, we show that the strong performance of label smoothing and Mixup on image classification benchmarks is correlated with learning low variance hidden representations. On the other hand, we show that Mixup and label smoothing can be more susceptible to low variance spurious correlations in the training data.