On the Generalization Properties of Adversarial Training
This addresses the problem of adversarial vulnerability in machine learning models, providing theoretical insights for improving robustness, though it is incremental by building on existing adversarial training frameworks.
The paper tackles the generalization performance of adversarial training algorithms, showing that in low-dimensional regimes, adversarial risk converges to minimal risk, while in high-dimensional regimes, L1 penalty leads to consistent robust estimation.
Modern machine learning and deep learning models are shown to be vulnerable when testing data are slightly perturbed. Existing theoretical studies of adversarial training algorithms mostly focus on either adversarial training losses or local convergence properties. In contrast, this paper studies the generalization performance of a generic adversarial training algorithm. Specifically, we consider linear regression models and two-layer neural networks (with lazy training) using squared loss under low-dimensional and high-dimensional regimes. In the former regime, after overcoming the non-smoothness of adversarial training, the adversarial risk of the trained models can converge to the minimal adversarial risk. In the latter regime, we discover that data interpolation prevents the adversarially robust estimator from being consistent. Therefore, inspired by successes of the least absolute shrinkage and selection operator (LASSO), we incorporate the L1 penalty in the high dimensional adversarial learning and show that it leads to consistent adversarially robust estimation. A series of numerical studies are conducted to demonstrate how the smoothness and L1 penalization help improve the adversarial robustness of DNN models.