Revisiting hard thresholding for DNN pruning
This work addresses the problem of efficient and accurate neural network pruning for practitioners, offering incremental improvements with a new algorithm and theoretical insights.
The paper tackles DNN pruning by showing that hard thresholding with retraining is the most efficient method for total pruning time, and proposes a novel smart pruning algorithm based on difference of convex functions optimization that is orders of magnitude faster with the lowest accuracy degradation. It also theoretically analyzes how accuracy degradation from hard thresholding increases with network depth and links it to the latent dimensionality of the training data manifold.
The most common method for DNN pruning is hard thresholding of network weights, followed by retraining to recover any lost accuracy. Recently developed smart pruning algorithms use the DNN response over the training set for a variety of cost functions to determine redundant network weights, leading to less accuracy degradation and possibly less retraining time. For experiments on the total pruning time (pruning time + retraining time) we show that hard thresholding followed by retraining remains the most efficient way of reducing the number of network parameters. However smart pruning algorithms still have advantages when retraining is not possible. In this context we propose a novel smart pruning algorithm based on difference of convex functions optimisation and show that it is often orders of magnitude faster than competing approaches while achieving the lowest classification accuracy degradation. Furthermore we investigate theoretically the effect of hard thresholding on DNN accuracy. We show that accuracy degradation increases with remaining network depth from the pruned layer. We also discover a link between the latent dimensionality of the training data manifold and network robustness to hard thresholding.