Symmetric Pruning of Large Language Models
This work addresses the problem of improving pruning efficiency and understanding for large language models, offering incremental advancements with practical gains.
The paper tackles the lack of theoretical foundation for post-training pruning methods like Wanda and RIA by introducing new theoretical insights and complementary strategies, resulting in a novel training-free fine-tuning approach that significantly outperforms baselines and sets a new state of the art.
Popular post-training pruning methods such as Wanda and RIA are known for their simple, yet effective, designs that have shown exceptional empirical performance. Wanda optimizes performance through calibrated activations during pruning, while RIA emphasizes the relative, rather than absolute, importance of weight elements. Despite their practical success, a thorough theoretical foundation explaining these outcomes has been lacking. This paper introduces new theoretical insights that redefine the standard minimization objective for pruning, offering a deeper understanding of the factors contributing to their success. Our study extends beyond these insights by proposing complementary strategies that consider both input activations and weight significance. We validate these approaches through rigorous experiments, demonstrating substantial enhancements over existing methods. Furthermore, we introduce a novel training-free fine-tuning approach $R^2$-DSnoT that incorporates relative weight importance and a regularized decision boundary within a dynamic pruning-and-growing framework, significantly outperforming strong baselines and establishing a new state of the art.