Understanding the effect of sparsity on neural networks robustness
This addresses the problem of understanding robustness trade-offs in sparsified neural networks for researchers and practitioners, showing that prior observed robustness drops are due to reduced capacity rather than sparsity itself, which is incremental.
The paper investigates how static sparsity affects neural network robustness to weight perturbations, data corruption, and adversarial examples, finding that sparsified networks can match or outperform dense versions up to a certain sparsity level, with robustness and accuracy declining only at very high sparsity due to loose connectivity.
This paper examines the impact of static sparsity on the robustness of a trained network to weight perturbations, data corruption, and adversarial examples. We show that, up to a certain sparsity achieved by increasing network width and depth while keeping the network capacity fixed, sparsified networks consistently match and often outperform their initially dense versions. Robustness and accuracy decline simultaneously for very high sparsity due to loose connectivity between network layers. Our findings show that a rapid robustness drop caused by network compression observed in the literature is due to a reduced network capacity rather than sparsity.