Data-Efficient Structured Pruning via Submodular Optimization
This addresses the challenge of compressing large models efficiently for scenarios with limited data, offering a principled solution with performance guarantees.
The paper tackles the problem of structured pruning of neural networks in data-limited settings by proposing a method based on submodular optimization, which guarantees exponentially decreasing error and outperforms state-of-the-art methods without requiring fine-tuning or labels.
Structured pruning is an effective approach for compressing large pre-trained neural networks without significantly affecting their performance. However, most current structured pruning methods do not provide any performance guarantees, and often require fine-tuning, which makes them inapplicable in the limited-data regime. We propose a principled data-efficient structured pruning method based on submodular optimization. In particular, for a given layer, we select neurons/channels to prune and corresponding new weights for the next layer, that minimize the change in the next layer's input induced by pruning. We show that this selection problem is a weakly submodular maximization problem, thus it can be provably approximated using an efficient greedy algorithm. Our method is guaranteed to have an exponentially decreasing error between the original model and the pruned model outputs w.r.t the pruned size, under reasonable assumptions. It is also one of the few methods in the literature that uses only a limited-number of training data and no labels. Our experimental results demonstrate that our method outperforms state-of-the-art methods in the limited-data regime.