Understanding the Detrimental Class-level Effects of Data Augmentation
This addresses a detrimental side effect of data augmentation for image classification practitioners, though it is incremental as it builds on prior observations.
The paper tackles the problem that data augmentation improves average accuracy in image classification but can harm individual class accuracy by up to 20% on ImageNet, and it finds that most affected classes are ambiguous or involve fine-grained distinctions, with simple class-conditional strategies improving performance on these classes.
Data augmentation (DA) encodes invariance and provides implicit regularization critical to a model's performance in image classification tasks. However, while DA improves average accuracy, recent studies have shown that its impact can be highly class dependent: achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much as 20% on ImageNet. There has been little progress in resolving class-level accuracy drops due to a limited understanding of these effects. In this work, we present a framework for understanding how DA interacts with class-level learning dynamics. Using higher-quality multi-label annotations on ImageNet, we systematically categorize the affected classes and find that the majority are inherently ambiguous, co-occur, or involve fine-grained distinctions, while DA controls the model's bias towards one of the closely related classes. While many of the previously reported performance drops are explained by multi-label annotations, our analysis of class confusions reveals other sources of accuracy degradation. We show that simple class-conditional augmentation strategies informed by our framework improve performance on the negatively affected classes.