FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees
This addresses the computational bottleneck in pruning neural networks for practitioners, though it appears incremental as it builds on existing DCA optimization methods.
The paper tackles the problem of costly retraining in DNN pruning by proposing FeTa, a fast pruning algorithm based on difference of convex functions optimization that requires little or no retraining, and provides theoretical analysis on generalization error growth for bounded perturbations. Experiments show the method is orders of magnitude faster than competing approaches.
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. We start by proposing a cheap pruning algorithm for fully connected DNN layers based on difference of convex functions (DC) optimisation, that requires little or no retraining. We then provide a theoretical analysis for the growth in the Generalization Error (GE) of a DNN for the case of bounded perturbations to the hidden layers, of which weight pruning is a special case. Our pruning method is orders of magnitude faster than competing approaches, while our theoretical analysis sheds light to previously observed problems in DNN pruning. Experiments on commnon feedforward neural networks validate our results.