Impact of Label Noise on Learning Complex Features
This addresses the challenge of enabling deep models to learn more representative features from real-world data, though it appears incremental as it builds on existing pre-training methods.
The paper tackles the problem of neural networks' inductive bias towards simple features by investigating the impact of pre-training with noisy labels, showing that it promotes learning complex and diverse features without performance loss.
Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and often fail to capture more complex features. This phenomenon raises concerns about the capacity of deep models to adequately learn and represent real-world datasets. Traditional approaches such as explicit regularization, data augmentation, architectural modifications, etc., have largely proven ineffective in encouraging the models to learn diverse features. In this work, we investigate the impact of pre-training models with noisy labels on the dynamics of SGD across various architectures and datasets. We show that pretraining promotes learning complex functions and diverse features in the presence of noise. Our experiments demonstrate that pre-training with noisy labels encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.