Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
This addresses the dual challenge of model compactness and adversarial robustness for deep learning practitioners, offering an incremental improvement over existing methods.
The paper tackles the problem of improving neural network robustness against adversarial attacks while reducing model size, showing that weight pruning can enhance robustness and proposing Robust Sparse Regularization (RSR) to achieve this, resulting in an 85% weight pruning with 0.68% and 8.72% accuracy improvements on clean and perturbed data for ResNet-18 on CIFAR-10.
Deep Neural Network (DNN) trained by the gradient descent method is known to be vulnerable to maliciously perturbed adversarial input, aka. adversarial attack. As one of the countermeasures against adversarial attack, increasing the model capacity for DNN robustness enhancement was discussed and reported as an effective approach by many recent works. In this work, we show that shrinking the model size through proper weight pruning can even be helpful to improve the DNN robustness under adversarial attack. For obtaining a simultaneously robust and compact DNN model, we propose a multi-objective training method called Robust Sparse Regularization (RSR), through the fusion of various regularization techniques, including channel-wise noise injection, lasso weight penalty, and adversarial training. We conduct extensive experiments across popular ResNet-20, ResNet-18 and VGG-16 DNN architectures to demonstrate the effectiveness of RSR against popular white-box (i.e., PGD and FGSM) and black-box attacks. Thanks to RSR, 85% weight connections of ResNet-18 can be pruned while still achieving 0.68% and 8.72% improvement in clean- and perturbed-data accuracy respectively on CIFAR-10 dataset, in comparison to its PGD adversarial training baseline.