Scholar Feed

Mixture-of-experts routing

Tutel

Tutel: Adaptive Mixture-of-Experts at Scale

Superseded baseline#7 of 1,370 most-superseded · first seen Jun 7, 2022

Superseded — cited as a baseline and beaten by newer methods

6 papers critique it · 1 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Tutel as a baseline.

existing MoE training systems, such as DeepSpeed-MoE~deepspeed-moe, DeepSpeed-TED~deepspeed-ted, and Tutel~hwang2023tutel, do not effectively address this shifted bottleneck, causing a memory explosion
— X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
“it still struggles with load imbalance due to static expert assignment”
— Accelerating MoE Model Inference with Expert Sharding
“their current capabilities are limited to manual configuration of the pipeline degree or heuristic search methods within a constrained search space”
— FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
“Tutel~tutel2022 provides efficient MoE dispatch and combine kernels with dynamic top-K routing and adaptive parallelism switching, but focuses on the dispatch kernel rather than end-to-end training strategy selection and does not cover attention-layer parallelization.”
— Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
“Tutel tutel and Comet zhang2025comet propose operator-level chunking, which partitions FFN computations into fine-grained tiles and partially overlaps dispatch/combine with the MoE layer through tile-level pipelining. However, as shown in Fig.~fig:motivationchunk, the overlap window is bounded by FFN computation, leaving residual communication exposed.”
— DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
“Tutel and OpenMoE+t5x are primarily designed for large-scale pretraining on hundreds of GPUs, restricting accessibility for groups with limited resources.”
— LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

Beaten on benchmarks

Head-to-head results where a newer method reports beating Tutel. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.