LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery
This addresses the challenge of deploying LLMs efficiently for users with limited computational resources, though it is incremental as it builds on existing pruning and LoRA techniques.
The paper tackles the problem of reducing computational costs in large language models (LLMs) by introducing LoRAShear, an efficient structured pruning method that reduces model size by 20% with only a 1.0% performance degradation, using one GPU within a couple of days.
Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear.