The Price of Implicit Bias in Adversarially Robust Generalization
This work addresses the problem of improving adversarial robustness in machine learning models, particularly for classification tasks, by analyzing implicit bias, but it appears incremental as it builds on existing robust ERM frameworks.
The paper investigates how the implicit bias of optimization in robust empirical risk minimization affects adversarial robustness and generalization, showing that it can significantly impact model robustness through optimization algorithms or architecture, with verification on synthetic data and deep neural networks.
We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization. In classification settings under adversarial perturbations with linear models, we study what type of regularization should ideally be applied for a given perturbation set to improve (robust) generalization. We then show that the implicit bias of optimization in robust ERM can significantly affect the robustness of the model and identify two ways this can happen; either through the optimization algorithm or the architecture. We verify our predictions in simulations with synthetic data and experimentally study the importance of implicit bias in robust ERM with deep neural networks.