Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying
This work addresses the need for more efficient fine-tuning methods in machine learning, particularly for large language models, though it is incremental as it builds upon existing LoRA techniques.
The paper tackled the problem of improving parameter efficiency in Low-rank Adaptation (LoRA) by introducing Tied-LoRA, which uses weight tying and selective training to reduce trainable parameters while maintaining performance, achieving comparable results to standard LoRA with only a fraction of the parameters at higher ranks.
We introduce Tied-LoRA, a novel paradigm leveraging weight tying and selective training to enhance the parameter efficiency of Low-rank Adaptation (LoRA). Our exploration encompasses different plausible combinations of parameter training and freezing, coupled with weight tying, aimed at identifying the optimal trade-off between performance and the count of trainable parameters. Across $5$ diverse tasks and two foundational language models with different parameter counts, our experiments provide comprehensive insights into the inherent trade-offs between efficiency and performance. Our findings reveal a specific Tied-LoRA configuration that distinguishes itself by showcasing comparable performance to LoRA across multiple tasks while utilizing only a fraction of the parameters employed by the standard LoRA method, particularly at elevated ranks. This underscores the efficacy of Tied-LoRA in achieving impressive results with significantly reduced model complexity.