Cyclical Pruning for Sparse Neural Networks
This addresses a specific limitation in pruning algorithms for neural networks, offering an incremental improvement for practitioners seeking efficient model compression.
The paper tackles the problem of weight recovery in neural network pruning by proposing cyclical pruning, a periodic schedule that allows erroneously pruned weights to recover in subsequent cycles, resulting in improved performance over existing methods, especially at high sparsity ratios.
Current methods for pruning neural network weights iteratively apply magnitude-based pruning on the model weights and re-train the resulting model to recover lost accuracy. In this work, we show that such strategies do not allow for the recovery of erroneously pruned weights. To enable weight recovery, we propose a simple strategy called \textit{cyclical pruning} which requires the pruning schedule to be periodic and allows for weights pruned erroneously in one cycle to recover in subsequent ones. Experimental results on both linear models and large-scale deep neural networks show that cyclical pruning outperforms existing pruning algorithms, especially at high sparsity ratios. Our approach is easy to tune and can be readily incorporated into existing pruning pipelines to boost performance.