HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks
This work addresses a specific bottleneck in parameter-efficient fine-tuning for multi-head attention models, offering an incremental improvement over existing methods.
The paper tackles the limitation of Low-Rank Adaptation (LoRA) in multi-head self-attention by proposing HoRA, which uses joint hypernetworks to enable cross-head information sharing, resulting in superior performance over LoRA and other methods with minimal parameter increase across language and vision benchmarks.
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) technique that adapts large pre-trained models by adding low-rank matrices to their weight updates. However, in the context of fine-tuning multi-head self-attention (MHA), LoRA has been employed to adapt each attention head separately, thereby overlooking potential synergies across different heads. To mitigate this issue, we propose a novel Hyper-shared Low-Rank Adaptation (HoRA) method, which utilizes joint hypernetworks to generate low-rank matrices across attention heads. By coupling their adaptation through a shared generator, HoRA encourages cross-head information sharing, and thus directly addresses the aforementioned limitation of LoRA. By comparing LoRA and HoRA through the lens of hierarchical mixture of experts, our theoretical findings reveal that the latter achieves superior sample efficiency to the former. Furthermore, through extensive experiments across diverse language and vision benchmarks, we demonstrate that HoRA outperforms LoRA and other PEFT methods while requiring only a marginal increase in the number of trainable parameters.