ZeroGrad : Mitigating and Explaining Catastrophic Overfitting in FGSM Adversarial Training
This addresses a computational bottleneck in adversarial training for deep neural networks, though it is an incremental improvement.
The paper tackled catastrophic overfitting in FGSM adversarial training by proposing to zero small input gradients when crafting attacks, achieving competitive adversarial accuracy on various datasets.
Making deep neural networks robust to small adversarial noises has recently been sought in many applications. Adversarial training through iterative projected gradient descent (PGD) has been established as one of the mainstream ideas to achieve this goal. However, PGD is computationally demanding and often prohibitive in case of large datasets and models. For this reason, single-step PGD, also known as FGSM, has recently gained interest in the field. Unfortunately, FGSM-training leads to a phenomenon called ``catastrophic overfitting," which is a sudden drop in the adversarial accuracy under the PGD attack. In this paper, we support the idea that small input gradients play a key role in this phenomenon, and hence propose to zero the input gradient elements that are small for crafting FGSM attacks. Our proposed idea, while being simple and efficient, achieves competitive adversarial accuracy on various datasets.