Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
This work addresses the scalability limitations of Mixture-of-Experts models for researchers and practitioners by proposing a new scaling dimension.
This paper introduces Mixture of Universal Experts (MOUE), a generalization of Mixture-of-Experts (MoE) that scales model capacity by converting depth into "virtual width" through expert reuse across layers. MOUE consistently outperforms MoE baselines by up to 1.3% and achieves up to 4.2% gains when progressively converting existing MoE checkpoints.
Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.