Pruning Large Language Models via Accuracy Predictor
This work addresses the problem of model size and deployment inefficiencies for users of large language models, offering an incremental improvement over manual pruning methods.
The paper tackles the challenge of compressing large language models by proposing a novel pruning approach that uses an accuracy predictor to automatically select optimal models, resulting in a 9.48% drop in perplexity on Wikitext2 and a 6.28% increase in average accuracy on MMLU compared to baselines.
Large language models(LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks. However, substantial model size poses challenges to training, inference, and deployment so that it is necessary to compress the model. At present, most model compression for LLMs requires manual design of pruning features, which has problems such as complex optimization pipeline and difficulty in retaining the capabilities of certain parts of the model.Therefore, we propose a novel pruning approach: firstly, a training set of a certain number of architecture-accuracy pairs is established, and then a non-neural model is trained as an accuracy predictor. Using the accuracy predictor to further optimize the search space and search, the optimal model can be automatically selected. Experiments show that our proposed approach is effective and efficient. Compared with the baseline, the perplexity(PPL) on Wikitext2 and PTB dropped by 9.48% and 5,76% respectively, and the average accuracy of MMLU increased by 6.28%.