RaSA: Rank-Sharing Low-Rank Adaptation
This addresses a bottleneck in parameter-efficient fine-tuning for LLMs, offering improvements in rigorous tasks like code and math, but is incremental as it builds on LoRA.
The paper tackles the limited expressive capacity of Low-rank Adaptation (LoRA) in fine-tuning large language models, particularly for code generation and mathematical reasoning, by introducing Rank-Sharing Low-Rank Adaptation (RaSA), which enhances performance without increasing parameters.
Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly in rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers. By forming a shared rank pool and applying layer-specific weighting, RaSA effectively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks. Code, data and scripts are available at: https://github.com/zwhe99/RaSA.