Enhance Diffusion to Improve Robust Generalization
This work addresses the challenge of improving robust generalization in adversarial training for deep neural networks, offering a method that enhances performance without additional computational burden, though it is incremental as it builds on the existing PGD-AT framework.
The paper tackled the problems of hyperparameter sensitivity and poor robust generalization in adversarial training (AT) by analyzing the Projected Gradient Descent AT framework through a stochastic differential equation, showing that robust generalization correlates with the learning rate to batch size ratio, and proposed Diffusion Enhanced Adversarial Training (DEAT) which theoretically and empirically outperforms PGD-AT with no extra computational cost.
Deep neural networks are susceptible to human imperceptible adversarial perturbations. One of the strongest defense mechanisms is \emph{Adversarial Training} (AT). In this paper, we aim to address two predominant problems in AT. First, there is still little consensus on how to set hyperparameters with a performance guarantee for AT research, and customized settings impede a fair comparison between different model designs in AT research. Second, the robustly trained neural networks struggle to generalize well and suffer from tremendous overfitting. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT). We approximate the dynamic of PGD-AT by a continuous-time Stochastic Differential Equation (SDE), and show that the diffusion term of this SDE determines the robust generalization. An immediate implication of this theoretical finding is that robust generalization is positively correlated with the ratio between learning rate and batch size. We further propose a novel approach, \emph{Diffusion Enhanced Adversarial Training} (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden. We theoretically show that DEAT obtains a tighter generalization bound than PGD-AT. Our empirical investigation is extensive and firmly attests that DEAT universally outperforms PGD-AT by a significant margin.