MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
This addresses task interference in multi-task learning for domains like NLP and vision, but it is incremental as it builds on existing LoRA methods.
The paper tackled the problem of task interference in multi-task learning with LoRA by proposing MTL-LoRA, which incorporates task-adaptive parameters to differentiate tasks and capture shared knowledge, resulting in outperforming LoRA and its variants on benchmarks like natural language understanding and image-text understanding with comparable or fewer parameters.
Parameter-efficient fine-tuning (PEFT) has been widely employed for domain adaptation, with LoRA being one of the most prominent methods due to its simplicity and effectiveness. However, in multi-task learning (MTL) scenarios, LoRA tends to obscure the distinction between tasks by projecting sparse high-dimensional features from different tasks into the same dense low-dimensional intrinsic space. This leads to task interference and suboptimal performance for LoRA and its variants. To tackle this challenge, we propose MTL-LoRA, which retains the advantages of low-rank adaptation while significantly enhancing MTL capabilities. MTL-LoRA augments LoRA by incorporating additional task-adaptive parameters that differentiate task-specific information and capture shared knowledge across various tasks within low-dimensional spaces. This approach enables pre-trained models to jointly adapt to different target domains with a limited number of trainable parameters. Comprehensive experimental results, including evaluations on public academic benchmarks for natural language understanding, commonsense reasoning, and image-text understanding, as well as real-world industrial text Ads relevance datasets, demonstrate that MTL-LoRA outperforms LoRA and its various variants with comparable or even fewer learnable parameters in MTL setting.