MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
This addresses a bottleneck in parameter-efficient fine-tuning for large language models, offering an incremental improvement over existing methods.
The paper tackles the representational bottleneck in LoRA for fine-tuning large language models by proposing MASA, a multi-A shared adaptation architecture that replaces the single down-projection matrix with multiple experts, achieving a 1.08-point improvement (1.84% relative) over standard LoRA on the MMLU benchmark with 59.62% average accuracy.
Low-Rank Adaptation (LoRA) has emerged as a dominant method in Parameter-Efficient Fine-Tuning (PEFT) for large language models, which augments the transformer layer with one down-projection $A$ and one up-projection $B$. However, LoRA's reliance on a single down-projection matrix ($A$) creates a representational bottleneck, as this solitary feature extractor is inherently insufficient for capturing the diverse signals required by complex tasks. This motivates our architectural shift to focus on enriching the feature adaptation to improve the downstream task adaptation ability. We propose MASA (Multi-$A$ Shared Adaptation), an architecture that implements a multi-$A$, single-$B$ structure where the multi-$A$ expert ensemble is asymmetrically shared across layers to ensure parameter efficiency. In MASA, these specialized experts capture diverse features, which are then integrated by a single, layer-specific $B$-matrix. The effectiveness and versatility of our method are validated through a comprehensive suite of experiments spanning multi-domain generalization, single-domain specialization, and multi-task reasoning. For example, on the MMLU benchmark, MASA achieves an average accuracy of 59.62%, outperforming the standard LoRA by 1.08 points (a relative improvement of 1.84%) with comparable learnable parameters of 0.52%.