DiffLoRA: Differential Low-Rank Adapters for Large Language Models
This work addresses parameter-efficient fine-tuning for NLP practitioners, but it is incremental as it builds on existing differential attention and LoRA techniques.
The authors tackled the problem of adapting the differential attention mechanism for parameter-efficient fine-tuning of large language models, resulting in mixed performance with a notable +11 point improvement over LoRA on HumanEval but generally falling short of other methods.
Differential Transformer has recently been proposed to improve performance in Transformer models by canceling out noise through a denoiser attention mechanism. In this work, we introduce DiffLoRA, a parameter-efficient adaptation of the differential attention mechanism, with low-rank adapters on both positive and negative attention terms. This approach retains the efficiency of LoRA while aiming to benefit from the performance gains of differential attention. We evaluate DiffLoRA across a broad range of NLP tasks, including general benchmarks, many-shot in-context learning, RAG, and long-context tests. We observe that, although DiffLoRA falls short of other parameter-efficient fine-tuning methods in most evaluation tasks, it shows interesting results in certain domains (+11 pts on LoRA for HumanEval). We analyze the attention patterns post-finetuning to identify the reasons for this behavior.