Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs
This addresses the problem of parameter-efficient fine-tuning applicability for researchers and practitioners working with diverse LLMs, though it is incremental as it builds on existing LoRA methods.
The paper tackles the limitation of LoRA being tied to specific base models by introducing Cross-LoRA, a data-free framework for transferring LoRA modules across heterogeneous LLMs, achieving relative gains of up to 5.26% on benchmarks like ARCs and OBOA.
Traditional parameter-efficient fine-tuning (PEFT) methods such as LoRA are tightly coupled with the base model architecture, which constrains their applicability across heterogeneous pretrained large language models (LLMs). To address this limitation, we introduce Cross-LoRA, a data-free framework for transferring LoRA modules between diverse base models without requiring additional training data. Cross-LoRA consists of two key components: (a) LoRA-Align, which performs subspace alignment between source and target base models through rank-truncated singular value decomposition (SVD) and Frobenius-optimal linear transformation, ensuring compatibility under dimension mismatch; and (b) LoRA-Shift, which applies the aligned subspaces to project source LoRA weight updates into the target model parameter space. Both components are data-free, training-free, and enable lightweight adaptation on a commodity GPU in 20 minutes. Experiments on ARCs, OBOA and HellaSwag show that Cross-LoRA achieves relative gains of up to 5.26% over base models. Across other commonsense reasoning benchmarks, Cross-LoRA maintains performance comparable to that of directly trained LoRA adapters.