Low-Rank Interconnected Adaptation across Layers
This work addresses a bottleneck in parameter-efficient fine-tuning for machine learning practitioners, offering an incremental improvement over existing methods.
The paper tackles the limitation of low-rank adaptation (LoRA) in parameter-efficient fine-tuning by proposing Lily, a method that uses interconnected experts to enable higher-rank weight updates with fewer parameters, achieving superior performance and efficiency across various modalities, architectures, and model sizes.
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates $ΔW = AB$ for pretrained weights $W$ through low-rank adapters $A$ and $B$. While LoRA ensures hardware efficiency, its low-rank weight updates limit adaptation performance. In this paper, we propose low-rank interconnected adaptation across layers (Lily), a novel PEFT method that introduces an interconnected framework with locally shared $A$ and globally shared $B$ experts. This structure eliminates redundant per-layer $AB$ pairs, enabling higher-rank $ΔW$ with equal or fewer parameters. To enhance expressiveness, we use data-dependent routers to determine $A$-$B$ interconnections, preventing $B$ experts from converging to the same behavior and improving representational power across domains. Experiments across modalities, architectures, and model sizes demonstrate Lily's superior performance and efficiency. GitHub: https://github.com/yibozhong/lily