Composition of Saliency Metrics for Channel Pruning with a Myopic Oracle
This addresses the challenge for data scientists in selecting effective pruning heuristics, though it is incremental as it builds on existing saliency metrics without a paradigm shift.
The paper tackles the problem of poor pruning decisions in convolutional neural networks due to reliance on a single saliency metric, by proposing a method to compose multiple saliency metrics to exploit their strengths. The result shows that this composition avoids many poor pruning choices and often outperforms the best individual saliency metric.
The computation and memory needed for Convolutional Neural Network (CNN) inference can be reduced by pruning weights from the trained network. Pruning is guided by a pruning saliency, which heuristically approximates the change in the loss function associated with the removal of specific weights. Many pruning signals have been proposed, but the performance of each heuristic depends on the particular trained network. This leaves the data scientist with a difficult choice. When using any one saliency metric for the entire pruning process, we run the risk of the metric assumptions being invalidated, leading to poor decisions being made by the metric. Ideally we could combine the best aspects of different saliency metrics. However, despite an extensive literature review, we are unable to find any prior work on composing different saliency metrics. The chief difficulty lies in combining the numerical output of different saliency metrics, which are not directly comparable. We propose a method to compose several primitive pruning saliencies, to exploit the cases where each saliency measure does well. Our experiments show that the composition of saliencies avoids many poor pruning choices identified by individual saliencies. In most cases our method finds better selections than even the best individual pruning saliency.