Expert Merging in Sparse Mixture of Experts with Nash Bargaining
This addresses the lack of principled weighting mechanisms in expert merging for SMoE, offering a novel game-theoretic approach with broad applicability across domains.
The paper tackles the problem of expert merging in Sparse Mixture of Experts (SMoE) by introducing Nash Merging of Experts (NAMEx), a framework that uses Nash Bargaining to enable more balanced and efficient collaboration among experts. The method consistently outperforms competing methods across language modeling, text classification, image classification, and zero-shot robustness tasks, and scales effectively to large models like Qwen1.5-MoE (14B) and DeepSeek-MoE (16B).
Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings.