Adversarial Risk Bounds for Neural Networks through Sparsity based Compression
This work addresses the generalization of adversarial robustness for neural networks, which is an incremental advance in theoretical machine learning.
The paper tackles the problem of adversarial robustness generalization in neural networks by proving margin-based bounds for ℓ∞ attacks on ℓ∞ bounded inputs, using a compression approach based on effective sparsity of weight matrices, resulting in bounds with no explicit dependence on input dimension or number of classes.
Neural networks have been shown to be vulnerable against minor adversarial perturbations of their inputs, especially for high dimensional data under $\ell_\infty$ attacks. To combat this problem, techniques like adversarial training have been employed to obtain models which are robust on the training set. However, the robustness of such models against adversarial perturbations may not generalize to unseen data. To study how robustness generalizes, recent works assume that the inputs have bounded $\ell_2$-norm in order to bound the adversarial risk for $\ell_\infty$ attacks with no explicit dimension dependence. In this work we focus on $\ell_\infty$ attacks on $\ell_\infty$ bounded inputs and prove margin-based bounds. Specifically, we use a compression based approach that relies on efficiently compressing the set of tunable parameters without distorting the adversarial risk. To achieve this, we apply the concept of effective sparsity and effective joint sparsity on the weight matrices of neural networks. This leads to bounds with no explicit dependence on the input dimension, neither on the number of classes. Our results show that neural networks with approximately sparse weight matrices not only enjoy enhanced robustness, but also better generalization.