EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning
For practitioners fine-tuning large MoE models, EPnG provides a parameter-efficient method that outperforms LoRA under the same budget, reducing resource costs while maintaining performance.
EPnG adaptively prunes under-utilized experts and grows high-importance experts via rank growth with orthogonal initialization, achieving performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer) on OLMoE and Qwen1.5-MoE.
Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.