Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
This addresses a bottleneck in parameter-efficient fine-tuning for large language models, though it appears incremental as it builds directly on LoRA.
The paper tackles the limitation of LoRA's zero-initialization bottleneck by proposing IniLoRA, a novel initialization strategy that approximates original model weights, achieving better performance than LoRA across various models and tasks.
The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread popularity due to its strong balance of effectiveness and parameter efficiency. However, LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. To address this limitation, we propose \textbf{IniLoRA}, a novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. Experimental results indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. Additionally, we introduce two variants, IniLoRA-$α$ and IniLoRA-$β$, both leveraging distinct initialization methods to enhance performance further.