The Search for Sparse, Robust Neural Networks
This work addresses the need for efficient and secure neural networks in safety-critical applications, representing an incremental advance by combining existing pruning and adversarial training methods.
The paper tackles the problem of finding sparse neural networks that are also robust to adversarial attacks, challenging the intuition of a trade-off between sparsity and robustness. Through empirical evaluation using the Lottery Ticket Hypothesis with adversarial training, they demonstrate that sparse, robust networks can be achieved, though no specific numerical results are provided in the abstract.
Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to pruning literature, deep neural networks are known to be susceptible to adversarial examples, which may pose risks in security- or safety-critical applications. Intuition suggests that there is an inherent trade-off between sparsity and robustness such that these characteristics could not co-exist. We perform an extensive empirical evaluation and analysis testing the Lottery Ticket Hypothesis with adversarial training and show this approach enables us to find sparse, robust neural networks. Code for reproducing experiments is available here: https://github.com/justincosentino/robust-sparse-networks.