Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100
This work addresses the challenge of improving deterministic l2 robustness for CNNs on image datasets like CIFAR-10 and CIFAR-100, representing an incremental advancement with specific gains.
The paper tackled the problem of training 1-Lipschitz convolutional neural networks for provable adversarial robustness by relaxing the orthogonalization of the last linear layer, achieving gains of 4.80% and 4.71% in standard and provable robust accuracies on CIFAR-100, and a 5.81% improvement in provable robust accuracy on CIFAR-10 with only a minor drop in standard accuracy.
Training convolutional neural networks (CNNs) with a strict Lipschitz constraint under the $l_{2}$ norm is useful for provable adversarial robustness, interpretable gradients and stable training. While $1$-Lipschitz CNNs can be designed by enforcing a $1$-Lipschitz constraint on each layer, training such networks requires each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent the gradients from vanishing during backpropagation. A layer with this property is said to be Gradient Norm Preserving (GNP). In this work, we introduce a procedure to certify the robustness of $1$-Lipschitz CNNs by relaxing the orthogonalization of the last linear layer of the network that significantly advances the state of the art for both standard and provable robust accuracies on CIFAR-100 (gains of $4.80\%$ and $4.71\%$, respectively). We further boost their robustness by introducing (i) a novel Gradient Norm preserving activation function called the Householder activation function (that includes every $\mathrm{GroupSort}$ activation) and (ii) a certificate regularization. On CIFAR-10, we achieve significant improvements over prior works in provable robust accuracy ($5.81\%$) with only a minor drop in standard accuracy ($-0.29\%$). Code for reproducing all experiments in the paper is available at \url{https://github.com/singlasahil14/SOC}.