Understanding Overfitting in Adversarial Training via Kernel Regression
This work addresses overfitting issues in adversarial training for machine learning practitioners, providing theoretical insights that are incremental to existing empirical observations.
The paper investigates adversarial training and noise-based data augmentation in kernel regression, showing that without proper regularization, these methods can increase generalization error and Lipschitz constant compared to standard kernel regression, but with appropriate regularization, they can outperform it by reducing these metrics.
Adversarial training and data augmentation with noise are widely adopted techniques to enhance the performance of neural networks. This paper investigates adversarial training and data augmentation with noise in the context of regularized regression in a reproducing kernel Hilbert space (RKHS). We establish the limiting formula for these techniques as the attack and noise size, as well as the regularization parameter, tend to zero. Based on this limiting formula, we analyze specific scenarios and demonstrate that, without appropriate regularization, these two methods may have larger generalization error and Lipschitz constant than standard kernel regression. However, by selecting the appropriate regularization parameter, these two methods can outperform standard kernel regression and achieve smaller generalization error and Lipschitz constant. These findings support the empirical observations that adversarial training can lead to overfitting, and appropriate regularization methods, such as early stopping, can alleviate this issue.