Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
This work addresses a foundational problem in AI theory for researchers and practitioners by offering novel theoretical insights into MoE optimization, though it is incremental as it builds on prior limited analyses.
The paper tackles the limited theoretical understanding of Mixture-of-Experts (MoE) training dynamics by providing convergence guarantees for joint training of soft-routed MoE models with non-linear routers and experts in a student-teacher framework, proving that the student network recovers the teacher's parameters with moderate over-parameterization and that post-training pruning and fine-tuning reach global optimality.
Mixture-of-Experts (MoE) architectures have emerged as a cornerstone of modern AI systems. In particular, MoEs route inputs dynamically to specialized experts whose outputs are aggregated through weighted summation. Despite their widespread application, theoretical understanding of MoE training dynamics remains limited to either separate expert-router optimization or only top-1 routing scenarios with carefully constructed datasets. This paper advances MoE theory by providing convergence guarantees for joint training of soft-routed MoE models with non-linear routers and experts in a student-teacher framework. We prove that, with moderate over-parameterization, the student network undergoes a feature learning phase, where the router's learning process is ``guided'' by the experts, that recovers the teacher's parameters. Moreover, we show that a post-training pruning can effectively eliminate redundant neurons, followed by a provably convergent fine-tuning process that reaches global optimality. To our knowledge, our analysis is the first to bring novel insights in understanding the optimization landscape of the MoE architecture.