A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs
This work provides theoretical insights into the geometry and trade-offs in adversarial training for high-dimensional linear models, which is incremental to existing robustness literature.
The authors tackled the problem of understanding adversarial training for linear classifiers in high dimensions by introducing a tractable statistical model that captures the interplay between data and attacker geometries. They derived exact asymptotic descriptions of the adversarial risk minimiser, revealing that multiple feature types are crucial for high performance and identifying directions that can be defended without sacrificing accuracy.
This work investigates adversarial training in the context of margin-based linear classifiers in the high-dimensional regime where the dimension $d$ and the number of data points $n$ diverge with a fixed ratio $α= n / d$. We introduce a tractable mathematical model where the interplay between the data and adversarial attacker geometries can be studied, while capturing the core phenomenology observed in the adversarial robustness literature. Our main theoretical contribution is an exact asymptotic description of the sufficient statistics for the adversarial empirical risk minimiser, under generic convex and non-increasing losses for a Block Feature Model. Our result allow us to precisely characterise which directions in the data are associated with a higher generalisation/robustness trade-off, as defined by a robustness and a usefulness metric. We show that the the presence of multiple different feature types is crucial to the high sample complexity performances of adversarial training. In particular, we unveil the existence of directions which can be defended without penalising accuracy. Finally, we show the advantage of defending non-robust features during training, identifying a uniform protection as an inherently effective defence mechanism.