Adversarial Robustness is at Odds with Lazy Training
This work highlights a fundamental trade-off between adversarial robustness and efficient lazy training in deep learning theory, which is incremental as it extends prior findings on random networks.
The paper demonstrates that over-parametrized neural networks trained under lazy training regimes, which are known to generalize well and have strong computational guarantees, remain vulnerable to adversarial attacks generated using a single step of gradient ascent.
Recent works show that adversarial examples exist for random neural networks [Daniely and Schacham, 2020] and that these examples can be found using a single step of gradient ascent [Bubeck et al., 2021]. In this work, we extend this line of work to "lazy training" of neural networks -- a dominant model in deep learning theory in which neural networks are provably efficiently learnable. We show that over-parametrized neural networks that are guaranteed to generalize well and enjoy strong computational guarantees remain vulnerable to attacks generated using a single step of gradient ascent.