Blockwise Hadamard high-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
This work addresses the problem of handling heterogeneous reasoning transformations in PEFT for large language models, offering an incremental improvement over existing methods like LoRA and HiRA.
The paper tackles the limitation of global modulation in parameter-efficient fine-tuning (PEFT) methods by proposing Block Hadamard high-Rank Adaptation (BHRA), which partitions weight matrices to apply localized rank amplification, and results show it consistently outperforms strong PEFT baselines across multiple tasks and models under matched parameter budgets.
Parameter-efficient fine-tuning (PEFT) methods must be resource-efficient yet handle heterogeneous reasoning transformations, and classical low-rank adaptation (LoRA) is constrained by the nominal rank $r$. Hadamard-style extensions like HiRA raise the nominal rank but couple every update to the global energy pattern of the frozen weight matrix, while ABBA trades this inductive bias for fully learned dense intermediates. To address the limitation of global modulation, we propose Block Hadamard high-Rank Adaptation (BHRA), which partitions each weight matrix and applies HiRA-style multiplicative modulation independently within every block, preserving the PEFT parameter footprint while unlocking localized rank amplification. Our empirical analyses reveal that this blockwise design maintains rich spectra across rank budgets, mitigating the collapse induced by global modulation. Across eight commonsense reasoning tasks and two arithmetic benchmarks with Llama-3.2 1B/3B, Mistral-7B, and Gemma-2 9B, BHRA consistently surpasses strong PEFT baselines under matched parameter budgets.