P$^2$ Law: Scaling Law for Post-Training After Model Pruning
This work addresses the cost-performance trade-off in deploying pruned LLMs, offering practical guidance for researchers and engineers, though it is incremental as it builds on existing pruning and post-training methods.
The paper tackles the problem of optimizing post-training data for pruned large language models by discovering the P^2 Law, which predicts post-training loss based on four factors, enabling efficient recovery of model performance with reduced data requirements.
Pruning has become a widely adopted technique for reducing the hardware requirements of large language models (LLMs). To recover model performance after pruning, post-training is commonly employed to mitigate the resulting performance degradation. While post-training benefits from larger datasets, once the dataset size is already substantial, increasing the training data provides only limited performance gains. To balance post-training cost and model performance, it is necessary to explore the optimal amount of post-training data.Through extensive experiments on the Llama-3 and Qwen-2.5 series models, pruned using various common pruning methods, we uncover the scaling \textbf{Law} for \textbf{P}ost-training after model \textbf{P}runing, referred to as the P$^2$ Law.This law identifies four key factors for predicting the pruned model's post-training loss: the model size before pruning, the number of post-training tokens, the pruning rate, and the model's loss before pruning. Moreover, P$^2$ Law can generalize to larger dataset sizes, larger model sizes, and higher pruning rates, offering valuable insights for the post-training of pruned LLMs.