PrunedLoRA: Robust Gradient-Based structured pruning for Low-rank Adaptation in Fine-tuning
This work addresses a key bottleneck in parameter-efficient fine-tuning for large language models, offering a novel method to enhance adapter expressiveness, though it is incremental within the LoRA paradigm.
The paper tackles the problem of low representational capacity in Low-rank Adaptation (LoRA) for fine-tuning large language models by proposing PrunedLoRA, a framework that uses structured pruning to dynamically allocate rank and improve expressiveness, achieving consistent performance gains over LoRA and its variants in tasks like mathematical reasoning, code generation, and natural language understanding.
Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning. Within the context of LoRA, a key open question is how to obtain expressive low-rank adapters from over-parameterized spaces. We propose \textit{PrunedLoRA}, a new framework that leverages structured pruning to obtain highly representative low-rank adapters from an over-parameterized initialization. Unlike prior approaches that impose a fixed low-rank budget, PrunedLoRA dynamically prunes less important components during fine-tuning and prevents their reactivation, enabling flexible and adaptive rank allocation. For structured pruning, by minimizing the pruning error for overall loss, we provide fine-grained pruning and recovery updates in a gradient-based pruning strategy with grounded interpretation. We provide the first theoretical analysis of the robustness of structured pruning and provably show that under the impact of weight perturbation, gradient-based pruning is more robust than activation-based pruning with respect to overall loss. Empirically, PrunedLoRA consistently outperforms LoRA and its variants across supervised fine-tuning tasks in mathematical reasoning, code generation, and natural language understanding, and it also demonstrates advantages over existing structured pruning methods across diverse sparsity levels.