Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism
This addresses the high cost of training multi-billion-parameter foundation models, making such research more accessible, though it is an incremental improvement on existing MoE methods.
The paper tackles the communication inefficiency and load imbalance in sparse Mixture of Experts (MoE) training for large language models by proposing Multi-Head LatentMoE and Head Parallel, achieving up to 1.61× faster training with identical performance.
Large language models have transformed many applications but remain expensive to train. Sparse Mixture of Experts (MoE) addresses this through conditional computation, with Expert Parallel (EP) as the standard distributed training method. However, EP has three limitations: communication cost grows linearly with the number of activated experts $k$, load imbalance affects latency and memory usage, and data-dependent communication requires metadata exchange. We propose Multi-Head LatentMoE and Head Parallel (HP), a new architecture and parallelism achieving $O(1)$ communication cost regardless of $k$, completely balanced traffic, and deterministic communication, all while remaining compatible with EP. To accelerate Multi-Head LatentMoE, we propose IO-aware routing and expert computation. Compared to MoE with EP, Multi-Head LatentMoE with HP trains up to $1.61\times$ faster while having identical performance. With doubled granularity, it achieves higher overall performance while still being $1.11\times$ faster. Our method makes multi-billion-parameter foundation model research more accessible.