Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness
This position paper highlights a critical limitation in adapter-based model merging for researchers and practitioners, suggesting that current approaches may be incremental without addressing fundamental effectiveness issues.
The paper argues that reusing low-rank adapters (LoRAs) often fails to achieve genuine compositional generalization, as shown through theoretical analysis and empirical tests on reasoning tasks, where methods like parameter averaging and dynamic selection struggled with integrating knowledge from disjoint datasets.
Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.