On the Tightness of Semidefinite Relaxations for Certifying Robustness to Adversarial Examples
This work addresses the need for tighter robustness certificates in adversarial machine learning, though it is incremental as it builds on existing SDP methods to analyze their limitations.
The paper tackles the problem of certifying neural network robustness to adversarial examples via semidefinite programming (SDP) relaxations, showing that the SDP certificate is exact for single hidden layers under mild assumptions but usually conservative for multiple layers, with experimental validation using interior-point and custom algorithms.
The robustness of a neural network to adversarial examples can be provably certified by solving a convex relaxation. If the relaxation is loose, however, then the resulting certificate can be too conservative to be practically useful. Recently, a less conservative robustness certificate was proposed, based on a semidefinite programming (SDP) relaxation of the ReLU activation function. In this paper, we describe a geometric technique that determines whether this SDP certificate is exact, meaning whether it provides both a lower-bound on the size of the smallest adversarial perturbation, as well as a globally optimal perturbation that attains the lower-bound. Concretely, we show, for a least-squares restriction of the usual adversarial attack problem, that the SDP relaxation amounts to the nonconvex projection of a point onto a hyperbola. The resulting SDP certificate is exact if and only if the projection of the point lies on the major axis of the hyperbola. Using this geometric technique, we prove that the certificate is exact over a single hidden layer under mild assumptions, and explain why it is usually conservative for several hidden layers. We experimentally confirm our theoretical insights using a general-purpose interior-point method and a custom rank-2 Burer-Monteiro algorithm.