Pruning via Iterative Ranking of Sensitivity Statistics
This work addresses the need for efficient and robust pruning methods in deep learning, offering a simpler alternative to existing approaches with broad applicability.
The paper tackles the problem of pruning neural networks before training by introducing SNIP-it, which applies sensitivity criteria iteratively in smaller steps, achieving state-of-the-art sparsity-performance trade-offs and demonstrating robustness to overfitting, disconnection, and adversarial attacks.
With the introduction of SNIP [arXiv:1810.02340v2], it has been demonstrated that modern neural networks can effectively be pruned before training. Yet, its sensitivity criterion has since been criticized for not propagating training signal properly or even disconnecting layers. As a remedy, GraSP [arXiv:2002.07376v1] was introduced, compromising on simplicity. However, in this work we show that by applying the sensitivity criterion iteratively in smaller steps - still before training - we can improve its performance without difficult implementation. As such, we introduce 'SNIP-it'. We then demonstrate how it can be applied for both structured and unstructured pruning, before and/or during training, therewith achieving state-of-the-art sparsity-performance trade-offs. That is, while already providing the computational benefits of pruning in the training process from the start. Furthermore, we evaluate our methods on robustness to overfitting, disconnection and adversarial attacks as well.