Adversarial Training is a Form of Data-dependent Operator Norm Regularization
This work addresses the problem of understanding and improving adversarial robustness in deep neural networks for researchers and practitioners, offering a foundational insight into the relationship between adversarial sensitivity and spectral properties.
The paper establishes a theoretical link between adversarial training and data-dependent operator norm regularization, proving that adversarial training with specific norms is equivalent to such regularization, and provides empirical evidence to support this connection.
We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks. Specifically, we prove that $\ell_p$-norm constrained projected gradient ascent based adversarial training with an $\ell_q$-norm loss on the logits of clean and perturbed inputs is equivalent to data-dependent (p, q) operator norm regularization. This fundamental connection confirms the long-standing argument that a network's sensitivity to adversarial examples is tied to its spectral properties and hints at novel ways to robustify and defend against adversarial attacks. We provide extensive empirical evidence on state-of-the-art network architectures to support our theoretical results.