Understanding the Interaction of Adversarial Training with Noisy Labels
This paper offers insights into how adversarial training can mitigate the effects of noisy labels, which is important for researchers and practitioners working on robust machine learning in real-world datasets.
The authors investigated the interaction between adversarial training (AT) and noisy labels (NL). They found that the number of PGD steps required to attack a point can indicate if it is mislabeled, and AT itself, due to its smoothing effects, is more robust to NL than standard training, leading to improved natural accuracy.
Noisy labels (NL) and adversarial examples both undermine trained models, but interestingly they have hitherto been studied independently. A recent adversarial training (AT) study showed that the number of projected gradient descent (PGD) steps to successfully attack a point (i.e., find an adversarial example in its proximity) is an effective measure of the robustness of this point. Given that natural data are clean, this measure reveals an intrinsic geometric property -- how far a point is from its class boundary. Based on this breakthrough, in this paper, we figure out how AT would interact with NL. Firstly, we find if a point is too close to its noisy-class boundary (e.g., one step is enough to attack it), this point is likely to be mislabeled, which suggests to adopt the number of PGD steps as a new criterion for sample selection for correcting NL. Secondly, we confirm AT with strong smoothing effects suffers less from NL (without NL corrections) than standard training (ST), which suggests AT itself is an NL correction. Hence, AT with NL is helpful for improving even the natural accuracy, which again illustrates the superiority of AT as a general-purpose robust learning criterion.