Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets
This addresses the challenge of deploying robust DNNs on resource-constrained devices like mobile phones, offering a novel integration of sparsity and robustness, though it is incremental as it builds on existing adversarial training and pruning methods.
The paper tackles the problem of compressing robustly trained deep neural networks (DNNs) by co-designing efficient compression algorithms and sparse architectures, achieving at least double the channel sparsity for adversarially trained ResNet20 on CIFAR10 while improving natural accuracy by 8.69% and robust accuracy under IFGSM attack by 5.42%.
Deep neural nets (DNNs) compression is crucial for adaptation to mobile devices. Though many successful algorithms exist to compress naturally trained DNNs, developing efficient and stable compression algorithms for robustly trained DNNs remains widely open. In this paper, we focus on a co-design of efficient DNN compression algorithms and sparse neural architectures for robust and accurate deep learning. Such a co-design enables us to advance the goal of accommodating both sparsity and robustness. With this objective in mind, we leverage the relaxed augmented Lagrangian based algorithms to prune the weights of adversarially trained DNNs, at both structured and unstructured levels. Using a Feynman-Kac formalism principled robust and sparse DNNs, we can at least double the channel sparsity of the adversarially trained ResNet20 for CIFAR10 classification, meanwhile, improve the natural accuracy by $8.69$\% and the robust accuracy under the benchmark $20$ iterations of IFGSM attack by $5.42$\%. The code is available at \url{https://github.com/BaoWangMath/rvsm-rgsm-admm}.