Learning to Prune in Training via Dynamic Channel Propagation
This addresses the problem of reducing computational costs in neural networks for practitioners, though it appears incremental as it builds on existing pruning methods.
The paper tackles the problem of pruning neural networks during training by introducing dynamic channel propagation, which selects channels based on utility and updates them via back-propagation, resulting in simultaneous training and pruning with verified superior performance on benchmark datasets like VGGNet and ResNet.
In this paper, we propose a novel network training mechanism called "dynamic channel propagation" to prune the neural networks during the training period. In particular, we pick up a specific group of channels in each convolutional layer to participate in the forward propagation in training time according to the significance level of channel, which is defined as channel utility. The utility values with respect to all selected channels are updated simultaneously with the error back-propagation process and will adaptively change. Furthermore, when the training ends, channels with high utility values are retained whereas those with low utility values are discarded. Hence, our proposed scheme trains and prunes neural networks simultaneously. We empirically evaluate our novel training scheme on various representative benchmark datasets and advanced convolutional neural network (CNN) architectures, including VGGNet and ResNet. The experiment results verify the superior performance and robust effectiveness of our approach.