Can Mixture-of-Experts Surpass Dense LLMs Under Strictly Equal Resources?
This addresses a practical problem for AI researchers and practitioners by clarifying the efficiency trade-offs of MoE architectures under fixed budgets, though it is incremental in refining existing methods.
The paper investigates whether Mixture-of-Experts (MoE) language models can outperform dense models under equal resource constraints, finding that with an optimal activation rate, MoEs surpass dense counterparts across different model sizes, validated by training over 250 models up to 7B scale and processing 50 trillion tokens.
Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource constraints - that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its significant practical value and potential. In this paper, we propose a novel perspective and methodological framework to study this question thoroughly. First, we comprehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data resource. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for the enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models at 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All models will be released publicly.