Protective Self-Adaptive Pruning to Better Compress DNNs
This work addresses the problem of efficiently compressing deep neural networks for practitioners, though it is incremental as it builds on existing adaptive pruning methods.
The paper tackles the high complexity and weak interpretability of adaptive network pruning methods by proposing Protective Self-Adaptive Pruning (PSAP), which uses weight sparsity ratios and a protective reconstruction mechanism to achieve superior accuracy and compression ratios on ImageNet and CIFAR-10, especially for high-ratio pruning.
Adaptive network pruning approach has recently drawn significant attention due to its excellent capability to identify the importance and redundancy of layers and filters and customize a suitable pruning solution. However, it remains unsatisfactory since current adaptive pruning methods rely mostly on an additional monitor to score layer and filter importance, and thus faces high complexity and weak interpretability. To tackle these issues, we have deeply researched the weight reconstruction process in iterative prune-train process and propose a Protective Self-Adaptive Pruning (PSAP) method. First of all, PSAP can utilize its own information, weight sparsity ratio, to adaptively adjust pruning ratio of layers before each pruning step. Moreover, we propose a protective reconstruction mechanism to prevent important filters from being pruned through supervising gradients and to avoid unrecoverable information loss as well. Our PSAP is handy and explicit because it merely depends on weights and gradients of model itself, instead of requiring an additional monitor as in early works. Experiments on ImageNet and CIFAR-10 also demonstrate its superiority to current works in both accuracy and compression ratio, especially for compressing with a high ratio or pruning from scratch.