Practical Convex Formulation of Robust One-hidden-layer Neural Network Training
This work addresses the challenge of adversarial robustness in neural networks for binary classification and regression, offering a more efficient alternative to current methods, though it is incremental as it builds on prior convex reformulation work.
The authors tackled the problem of training robust one-hidden-layer neural networks by developing a convex optimization approach that approximates an exact but exponentially large formulation with linear complexity, achieving noticeably better adversarial robustness and performance than existing methods like FGSM and PGD.
Recent work has shown that the training of a one-hidden-layer, scalar-output fully-connected ReLU neural network can be reformulated as a finite-dimensional convex program. Unfortunately, the scale of such a convex program grows exponentially in data size. In this work, we prove that a stochastic procedure with a linear complexity well approximates the exact formulation. Moreover, we derive a convex optimization approach to efficiently solve the "adversarial training" problem, which trains neural networks that are robust to adversarial input perturbations. Our method can be applied to binary classification and regression, and provides an alternative to the current adversarial training methods, such as Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). We demonstrate in experiments that the proposed method achieves a noticeably better adversarial robustness and performance than the existing methods.