NoiLIn: Improving Adversarial Training and Correcting Stereotype of Noisy Labels
This addresses the robustness-accuracy trade-off in adversarial training for machine learning models, offering a novel perspective on using noisy labels beneficially, though it is incremental as it builds on existing AT methods.
The paper tackles the problem of adversarial training (AT) suffering from robust overfitting and the robustness-accuracy trade-off by proposing NoiLIn, a method that injects noisy labels during training, which significantly mitigates robust overfitting and improves generalization in state-of-the-art AT methods.
Adversarial training (AT) formulated as the minimax optimization problem can effectively enhance the model's robustness against adversarial attacks. The existing AT methods mainly focused on manipulating the inner maximization for generating quality adversarial variants or manipulating the outer minimization for designing effective learning objectives. However, empirical results of AT always exhibit the robustness at odds with accuracy and the existence of the cross-over mixture problem, which motivates us to study some label randomness for benefiting the AT. First, we thoroughly investigate noisy labels (NLs) injection into AT's inner maximization and outer minimization, respectively and obtain the observations on when NL injection benefits AT. Second, based on the observations, we propose a simple but effective method -- NoiLIn that randomly injects NLs into training data at each training epoch and dynamically increases the NL injection rate once robust overfitting occurs. Empirically, NoiLIn can significantly mitigate the AT's undesirable issue of robust overfitting and even further improve the generalization of the state-of-the-art AT methods. Philosophically, NoiLIn sheds light on a new perspective of learning with NLs: NLs should not always be deemed detrimental, and even in the absence of NLs in the training set, we may consider injecting them deliberately. Codes are available in https://github.com/zjfheart/NoiLIn.