DR-LoRA: Dynamic Rank LoRA for Mixture-of-Experts Adaptation
This work addresses a resource mismatch issue in adapting large MoE models for downstream tasks, offering an incremental improvement over existing parameter-efficient fine-tuning techniques.
The paper tackles the problem of inefficient parameter allocation in fine-tuning Mixture-of-Experts (MoE) models by proposing DR-LoRA, a dynamic rank LoRA framework that adapts expert ranks based on task-specific demands, resulting in superior performance and more efficient parameter utilization compared to standard methods.
Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning (PEFT), such as LoRA, is widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches assign identical LoRA ranks to all experts, overlooking the intrinsic functional specialization within MoE LLMs. This uniform allocation leads to resource mismatch, task-relevant experts are under-provisioned while less relevant ones receive redundant parameters. We propose a Dynamic Rank LoRA framework named DR-LoRA, which dynamically grows expert LoRA ranks during fine-tuning based on task-specific demands. DR-LoRA employs an Expert Saliency Scoring mechanism that integrates expert routing frequency and LoRA rank importance to quantify each expert's demand for additional capacity. Experts with higher saliency scores are prioritized for rank expansion, enabling the automatic formation of a heterogeneous rank distribution tailored to the target task. Experiments on multiple benchmarks demonstrate that DR-LoRA consistently outperforms standard LoRA and static allocation strategies under the same parameter budget, achieving superior task performance with more efficient parameter utilization.