Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
This work addresses the challenge of efficiently designing MoE architectures for large language models, providing practitioners with quantitative flexibility to balance scaling recommendations against infrastructure constraints, though it is incremental in refining existing scaling approaches.
The paper tackles the problem of optimizing Mixture-of-Experts (MoE) architectures by developing a holistic framework that bridges scaling laws with precise configurations, establishing a joint constraint triad of FLOPs per token, active parameters, and total parameters to address inadequacies in existing metrics, and it yields robust scaling laws validated across hundreds of models spanning six orders of magnitude in compute.
Scaling laws for Large Language Models govern macroscopic resource allocation, yet translating them into precise Mixture-of-Experts (MoE) architectural configurations remains an open problem due to the combinatorially vast design space. Existing MoE scaling studies are constrained by experimental budgets to either augment scaling formulas with extra MoE variables, risking unreliable fits, or fix all non-MoE factors, ignoring global interactions. We propose a reusable framework for holistic MoE architectural optimization that bridges this gap. We first show that FLOPs per token alone is an inadequate fairness metric for MoE models because differing computational densities across layer types can inflate parameters without proportional compute cost, and establish a joint constraint triad of FLOPs per token, active parameters, and total parameters. We then reduce the 16-dimensional architectural search space to two sequential low-dimensional phases through algebraic constraints and a rank-preserving property of the hidden dimension. Validated across hundreds of MoE models spanning six orders of magnitude in compute, our framework yields robust scaling laws that map any compute budget to a complete, optimal MoE architecture. A key finding is that the near-optimal configuration band widens with scale, giving practitioners quantitative flexibility to balance scaling law recommendations against infrastructure constraints.