Align, Don't Divide: Revisiting the LoRA Architecture in Multi-Task Learning
This work addresses the problem of efficiently adapting large language models to multiple tasks, offering a simpler and more effective paradigm for researchers and practitioners, though it is incremental in refining existing PEFT methods.
The paper challenges the trend of using complex multi-adapter LoRA variants for multi-task learning in LLMs, showing that a simplified single-adapter LoRA with increased rank or an explicit alignment loss (Align-LoRA) achieves superior performance, with Align-LoRA significantly outperforming all baselines.
Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models (LLMs). In practice, LLMs are often required to handle a diverse set of tasks from multiple domains, a scenario naturally addressed by multi-task learning (MTL). Within this MTL context, a prevailing trend involves LoRA variants with multiple adapters or heads, which advocate for structural diversity to capture task-specific knowledge. Our findings present a direct challenge to this paradigm. We first show that a simplified multi-head architecture with high inter-head similarity substantially outperforms complex multi-adapter and multi-head systems. This leads us to question the multi-component paradigm itself, and we further demonstrate that a standard single-adapter LoRA, with a sufficiently increased rank, also achieves highly competitive performance. These results lead us to a new hypothesis: effective MTL generalization hinges on learning robust shared representations, not isolating task-specific features. To validate this, we propose Align-LoRA, which incorporates an explicit loss to align task representations within the shared adapter space. Experiments confirm that Align-LoRA significantly surpasses all baselines, establishing a simpler yet more effective paradigm for adapting LLMs to multiple tasks. The code is available at https://github.com/jinda-liu/Align-LoRA.