Scaling Adversarial Training via Data Selection
This addresses scalability issues in adversarial training for machine learning practitioners, offering a more efficient method without sacrificing robustness, though it is incremental as it builds on existing PGD approaches.
The paper tackled the high computational cost of Projected Gradient Descent (PGD) adversarial training by proposing Selective Adversarial Training, which perturbs only critical samples in each minibatch, achieving comparable or better robustness on MNIST and CIFAR-10 while reducing adversarial computation by up to 50%.
Projected Gradient Descent (PGD) is a strong and widely used first-order adversarial attack, yet its computational cost scales poorly, as all training samples undergo identical iterative inner-loop optimization despite contributing unequally to robustness. Motivated by this inefficiency, we propose \emph{Selective Adversarial Training}, which perturbs only a subset of critical samples in each minibatch. Specifically, we introduce two principled selection criteria: (1) margin-based sampling, which prioritizes samples near the decision boundary, and (2) gradient-matching sampling, which selects samples whose gradients align with the dominant batch optimization direction. Adversarial examples are generated only for the selected subset, while the remaining samples are trained cleanly using a mixed objective. Experiments on MNIST and CIFAR-10 show that the proposed methods achieve robustness comparable to, or even exceeding, full PGD adversarial training, while reducing adversarial computation by up to $50\%$, demonstrating that informed sample selection is sufficient for scalable adversarial robustness.