Benign Overfitting in Adversarially Robust Linear Classification
This addresses the robustness of overfitting in adversarial settings, providing theoretical justification for practitioners in security-critical domains, though it is incremental as it extends prior work to adversarial contexts.
The paper investigates whether benign overfitting persists in adversarial training for linear classification, showing that under moderate perturbations, adversarially trained classifiers achieve near-optimal standard and adversarial risks despite overfitting noisy data.
"Benign overfitting", where classifiers memorize noisy training data yet still achieve a good generalization performance, has drawn great attention in the machine learning community. To explain this surprising phenomenon, a series of works have provided theoretical justification in over-parameterized linear regression, classification, and kernel methods. However, it is not clear if benign overfitting still occurs in the presence of adversarial examples, i.e., examples with tiny and intentional perturbations to fool the classifiers. In this paper, we show that benign overfitting indeed occurs in adversarial training, a principled approach to defend against adversarial examples. In detail, we prove the risk bounds of the adversarially trained linear classifier on the mixture of sub-Gaussian data under $\ell_p$ adversarial perturbations. Our result suggests that under moderate perturbations, adversarially trained linear classifiers can achieve the near-optimal standard and adversarial risks, despite overfitting the noisy training data. Numerical experiments validate our theoretical findings.