HMoE: Heterogeneous Mixture of Experts for Language Modeling
This work addresses computational efficiency and parameter utilization in large language models, representing an incremental improvement over existing MoE methods.
The paper tackles the inefficiency of homogeneous Mixture of Experts (MoE) models by proposing HMoE, a heterogeneous version with experts of varying sizes, which achieves lower loss with fewer activated parameters and outperforms conventional MoE on pre-training benchmarks.
Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying complexity in input data necessitates experts with diverse capabilities, while homogeneous MoE hinders effective expert specialization and efficient parameter utilization. In this study, we propose a novel Heterogeneous Mixture of Experts (HMoE), where experts differ in size and thus possess diverse capacities. This heterogeneity allows for more specialized experts to handle varying token complexities more effectively. To address the imbalance in expert activation, we propose a novel training objective that encourages the frequent activation of smaller experts, enhancing computational efficiency and parameter utilization. Extensive experiments demonstrate that HMoE achieves lower loss with fewer activated parameters and outperforms conventional homogeneous MoE models on various pre-training evaluation benchmarks. Codes will be released upon acceptance.