Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
This addresses a critical bottleneck in scaling MoE models efficiently for AI/ML applications, offering a substantial improvement over existing methods.
The paper tackles the problem of severe load imbalance in Mixture-of-Experts (MoE) architectures for large language models, where only a few experts are activated, causing underutilization. It proposes Latent Prototype Routing (LPR), which reduces the Gini coefficient from 0.70 to 0.035 on average and improves the min-max expert load ratio from 1e-6 to 0.70, achieving near-perfect load balancing.
Mixture-of-Experts (MoE) architectures have emerged as a key strategy for scaling large language models (LLMs) efficiently. However, current MoE systems suffer from severe load imbalance, where only a small subset of experts is consistently activated during training and inference, leading to significant underutilization of model capacity and computational resources. In this work, we revisit expert routing through a clustering perspective and propose Latent Prototype Routing (LPR), a novel routing framework that generalizes existing approaches while promoting balanced expert utilization without compromising downstream performance. Extensive experiments across multiple open-source MoE models -- including DeepSeek-V3, Qwen3-MoE, and Mixtral -- demonstrate that LPR reduces the Gini coefficient of expert load from 0.70 to 0.035 on average, improves the min-max expert load ratio from 1e-6 to 0.70, achieving near-perfect load balancing.