Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
This work addresses a bottleneck in fine-tuning large MoE models for researchers and practitioners, offering incremental improvements by optimizing PEFT strategies specifically for MoE architectures.
The paper tackles the problem that existing Parameter-Efficient Fine-Tuning (PEFT) strategies do not leverage the dynamic routing in Mixture-of-Experts (MoE) models, and it shows that incorporating routing mechanisms into adaptation modules improves performance and efficiency on commonsense and math reasoning tasks, validated through experiments on models like OLMoE-1B-7B and Mixtral-8x7B.
Mixture-of-Experts (MoE) benefits from a dynamic routing mechanism among their specialized experts, which existing Parameter- Efficient Fine-Tuning (PEFT) strategies fail to leverage. This motivates us to investigate whether adaptation modules themselves should incorporate routing mechanisms to align with MoE's multi-expert architecture. We analyze dynamics of core components when applying PEFT to MoE language models and examine how different routing strategies affect adaptation effectiveness. Extensive experiments adapting OLMoE-1B-7B and Mixtral-8x7B on various commonsense and math reasoning tasks validate the performance and efficiency of our routed approach. We identify the optimal configurations for different scenarios and provide empirical analyses with practical insights to facilitate better PEFT and MoE applications.