ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning
This addresses the need for scalable and efficient fine-tuning in resource-constrained environments, offering an incremental improvement over fixed-rank LoRA methods.
The paper tackled the problem of inefficient rank allocation in Low-Rank Adaptation (LoRA) for fine-tuning large models by proposing ElaLoRA, which dynamically prunes and expands ranks based on gradient importance, resulting in consistent outperformance of existing PEFT methods across benchmarks.
Low-Rank Adaptation (LoRA) has become a widely adopted technique for fine-tuning large-scale pre-trained models with minimal parameter updates. However, existing methods rely on fixed ranks or focus solely on either rank pruning or expansion, failing to adapt ranks dynamically to match the importance of different layers during training. In this work, we propose ElaLoRA, an adaptive low-rank adaptation framework that dynamically prunes and expands ranks based on gradient-derived importance scores. To the best of our knowledge, ElaLoRA is the first method that enables both rank pruning and expansion during fine-tuning. Experiments across multiple benchmarks demonstrate that ElaLoRA consistently outperforms existing PEFT methods across different parameter budgets. Furthermore, our studies validate that layers receiving higher rank allocations contribute more significantly to model performance, providing theoretical justification for our adaptive strategy. By introducing a principled and adaptive rank allocation mechanism, ElaLoRA offers a scalable and efficient fine-tuning solution, particularly suited for resource-constrained environments.