HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
For practitioners using parameter-efficient fine-tuning of large language models, HiP-LoRA provides a robust method to mitigate catastrophic forgetting and improve multi-adapter merging.
HiP-LoRA addresses spectral interference in LoRA-based fine-tuning by decomposing updates into principal and residual channels, using a stability budget to balance preservation and plasticity. On Llama-3.1-8B, it reduces pretraining degradation and multi-adapter MergeFail, outperforming baselines in continual tuning and knowledge editing.
Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.