Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
For researchers working on efficient Transformer scaling, this work addresses the problem of unstable and inconsistent expert selection in MoE models.
The authors identify context incompleteness as a key bottleneck in Mixture-of-Experts routing and propose MCF-MOE, which integrates cross-layer semantic aggregation and local token-level interactions to improve routing consistency and downstream performance on language modeling and understanding benchmarks.
Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts. However, existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers. In this work, we revisit expert selection from a representation perspective and identify context incompleteness as a key bottleneck limiting effective expert specialization. To address this issue, we propose Multi-level Context Fusion MOE (MCF-MOE), a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions, enabling more informative and consistent expert selection. Experiments on language modeling and understanding benchmarks demonstrate that MCF-MOE consistently improves routing consistency and downstream performance over strong MoE baselines, highlighting the importance of contextual completeness in expert routing. The code is available at https://anonymous.4open.science/r/MCFMOE.