Structured Directional Pruning via Perturbation Orthogonal Projection
This addresses the need for efficient neural network compression with minimal accuracy loss, though it appears incremental as it builds on existing structured pruning techniques.
The paper tackles the problem of structured pruning in neural networks by proposing a method that finds sparse minimizers along flat minimum valleys to keep training loss constant, achieving state-of-the-art pruned accuracy of 93.97% on VGG16 for CIFAR-10 without retraining.
Structured pruning is an effective compression technique to reduce the computation of neural networks, which is usually achieved by adding perturbations to reduce network parameters at the cost of slightly increasing training loss. A more reasonable approach is to find a sparse minimizer along the flat minimum valley found by optimizers, i.e. stochastic gradient descent, which keeps the training loss constant. To achieve this goal, we propose the structured directional pruning based on orthogonal projecting the perturbations onto the flat minimum valley. We also propose a fast solver sDprun and further prove that it achieves directional pruning asymptotically after sufficient training. Experiments using VGG-Net and ResNet on CIFAR-10 and CIFAR-100 datasets show that our method obtains the state-of-the-art pruned accuracy (i.e. 93.97% on VGG16, CIFAR-10 task) without retraining. Experiments using DNN, VGG-Net and WRN28X10 on MNIST, CIFAR-10 and CIFAR-100 datasets demonstrate our method performs structured directional pruning, reaching the same minimum valley as the optimizer.