Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
This addresses the computational inefficiency of pruning in deep CNNs for practitioners, though it is an incremental improvement over existing pruning methods.
The paper tackles the problem of inefficient redundancy distribution in CNNs by proposing Centripetal SGD (C-SGD) to manipulate filters into identical patterns, enabling pruning without accuracy drops. Results on CIFAR and ImageNet show better performance and efficiency, with no need for finetuning and applicability to deep networks.
The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops. However, the training objective of CNNs usually tends to minimize an accuracy-related loss function without any attention paid to the redundancy, making the redundancy distribute randomly on all the filters, such that removing any of them may trigger information loss and accuracy drop, necessitating a following finetuning step for recovery. In this paper, we propose to manipulate the redundancy during training to facilitate network pruning. To this end, we propose a novel Centripetal SGD (C-SGD) to make some filters identical, resulting in ideal redundancy patterns, as such filters become purely redundant due to their duplicates; hence removing them does not harm the network. As shown on CIFAR and ImageNet, C-SGD delivers better performance because the redundancy is better organized, compared to the existing methods. The efficiency also characterizes C-SGD because it is as fast as regular SGD, requires no finetuning, and can be conducted simultaneously on all the layers even in very deep CNNs. Besides, C-SGD can improve the accuracy of CNNs by first training a model with the same architecture but wider layers then squeezing it into the original width.