GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods
This provides a simple, strong baseline for model pruning research, benefiting practitioners by highlighting the importance of parameter tuning.
The paper tackled the problem of pruning large language models like BERT by revisiting gradual magnitude pruning (GMP), showing that a well-tuned variant (GMP*) can match or outperform more complex state-of-the-art methods on various tasks.
We revisit the performance of the classic gradual magnitude pruning (GMP) baseline for large language models, focusing on the classic BERT benchmark on various popular tasks. Despite existing evidence in the literature that GMP performs poorly, we show that a simple and general variant, which we call GMP*, can match and sometimes outperform more complex state-of-the-art methods. Our results provide a simple yet strong baseline for future work, highlight the importance of parameter tuning for baselines, and even improve the performance of the state-of-the-art second-order pruning method in this setting.