HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
This addresses performance bottlenecks in fine-tuning large language models for complex tasks like mathematics and code generation, offering a significant improvement over existing methods.
The paper tackles the limited expressiveness of low-rank PEFT methods like LoRA and PiSSA for LLMs by introducing HD-PiSSA, a distributed approach that assigns orthogonal adapters across GPUs to expand update directions, achieving over 16x higher effective ranks and average gains of 10.0 points over LoRA and 4.98 points over PiSSA on multi-task benchmarks.
Existing parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), such as LoRA and PiSSA, constrain model updates to low-rank subspaces, limiting their expressiveness and leading to suboptimal performance on complex tasks. To address this, we introduce High-rank Distributed PiSSA (HD-PiSSA), a distributed PEFT approach that initializes orthogonal adapters across different devices and aggregates their delta updates collectively on W for fine-tuning. Unlike Data Parallel LoRA or PiSSA, which maintain identical adapters across all devices, HD-PiSSA assigns different principal components of the pre-trained weights to each GPU, significantly expanding the range of update directions. This results in over 16x higher effective updated ranks than data-parallel LoRA or PiSSA when fine-tuning on 8 GPUs with the same per-device adapter rank. Empirically, we evaluate HD-PiSSA across various challenging downstream tasks, including mathematics, code generation, and multi-task learning. In the multi-task setting, HD-PiSSA achieves average gains of 10.0 absolute points (14.63%) over LoRA and 4.98 points (6.60%) over PiSSA across 12 benchmarks, demonstrating its benefits from the extra optimization flexibility.