DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
For large language model practitioners, DAG-MoE offers a new way to scale MoE performance without increasing routing overhead, addressing a key bottleneck in fine-grained expert models.
DAG-MoE introduces structural aggregation of expert outputs in MoE models, replacing weighted summation with a learned directed acyclic graph to expand expert combinations and enable multi-step reasoning within a single layer, consistently improving performance in language modeling pretraining and fine-tuning over traditional MoE baselines.
Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge. Prior work shows that fine-grained experts enlarge the space of expert combinations and improve flexibility, but they also impose substantial routing overhead, creating a new scalability bottleneck. In this paper, we explore a complementary axis for scaling -- how expert outputs are aggregated. We theoretically show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router, and enables possible multi-step reasoning within a single MoE layer. To this end, we propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts. Extensive experiments under standard language modeling settings show that DAG-MoE consistently improves performance in both pretraining and fine-tuning, surpassing traditional MoE baselines.