ABM-LoRA: Activation Boundary Matching for Fast Convergence in Low-Rank Adaptation
This addresses faster and more efficient fine-tuning for large models, but it is incremental as it builds on existing LoRA methods.
The paper tackled the problem of slow convergence in low-rank adapters (LoRA) due to random initialization, proposing ABM-LoRA, an initialization strategy that aligns activation boundaries to accelerate training. It achieved the highest accuracy on VTAB-1K and strong gains on tasks requiring geometric understanding.
We propose Activation Boundary Matching for Low-Rank Adaptation (ABM-LoRA), a principled initialization strategy that substantially accelerates the convergence of low-rank adapters. While LoRA offers high parameter efficiency, its random initialization restricts gradient updates to a mismatched tangent space, causing significant information loss and hindering early convergence. Our ABM-LoRA addresses this by aligning the adapter's activation boundaries with those of the pretrained model before downstream training, thereby maximizing the projection of full-parameter gradients into the adapter subspace. This alignment sharply reduces information loss at initialization, yields a lower starting loss, and accelerates convergence. We demonstrate ABM-LoRA's effectiveness across diverse architectures and tasks: language understanding (T5-Base on GLUE), dialogue generation (LLaMA2-7B on WizardLM), and vision recognition (ViT-B/16 on VTAB-1K). On VTAB-1K, it achieves the highest accuracy among all methods, with strong gains on structured reasoning tasks requiring geometric understanding.