Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuning
This work addresses the need for efficient model compression in deep learning by reducing computational costs without fine-tuning, though it is incremental as it builds on existing pruning techniques.
The paper tackles the problem of resource-intensive pruning methods by proposing a retraining-free pruning approach using hyperspherical learning and loss penalty terms, achieving a 0.47% accuracy drop for a 50% pruned ResNet-18 model and a 3.5% drop for a 70% pruned MobileNetV2 model, which is significantly better than conventional methods.
Most existing pruning works are resource-intensive, requiring retraining or fine-tuning of the pruned models for accuracy. We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms. The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero and can be safely pruned with no need for retraining and a negligible accuracy drop. In addition, our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value. Our method obtains state-of-the-art results in retraining-free pruning and is evaluated on ResNet-18/50 and MobileNetV2 with ImageNet dataset. One can easily get a 50\% pruned ResNet18 model with a 0.47\% accuracy drop. With fine-tuning, the experiment results show that our method can significantly boost the accuracy of the pruned models compared with existing works. For example, the accuracy of a 70\% pruned (except the first convolutional layer) MobileNetV2 model only drops 3.5\%, much less than the 7\% $\sim$ 10\% accuracy drop with conventional methods.