Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Better
This work addresses the trade-off between robustness and generalization in adversarial training for machine learning practitioners, offering incremental improvements based on existing methods.
The paper tackles the problem that adversarial training (AT) often reduces generalization ability, and finds that leveraging clean training can improve AT by alleviating difficulties in learning certain sample features, leading to enhanced performance.
Adversarial training (AT) is an effective technique for enhancing adversarial robustness, but it usually comes at the cost of a decline in generalization ability. Recent studies have attempted to use clean training to assist adversarial training, yet there are contradictions among the conclusions. We comprehensively summarize the representative strategies and, with a focus on the multi - view hypothesis, provide a unified explanation for the contradictory phenomena among different studies. In addition, we conduct an in - depth analysis of the knowledge combinations transferred from clean - trained models to adversarially - trained models in previous studies, and find that they can be divided into two categories: reducing the learning difficulty and providing correct guidance. Based on this finding, we propose a new idea of leveraging clean training to further improve the performance of advanced AT methods.We reveal that the problem of generalization degradation faced by AT partly stems from the difficulty of adversarial training in learning certain sample features, and this problem can be alleviated by making full use of clean training.