Is Switch Transformer superseded?

Switch Transformer (Mixture-of-experts routing): heavily superseded — a standard baseline that newer methods routinely beat. 5 paper(s) critique it, 7 beat it on benchmarks — #1 of 1370 most-superseded. Sub-problem: cluster led by Switch Transformer. Newer alternatives in the same sub-problem include ConceptM³oE, DisagMoE, Piper, GRACE-MoE, ReaLB.

Mixture-of-experts routing

Switch Transformer

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Heavily superseded#1 of 1,370 most-superseded · first seen Jan 11, 2021

Heavily superseded — a standard baseline that newer methods routinely beat

5 papers critique it · 7 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Switch Transformer as a baseline.

While these issues are well explored, in this paper we highlight an under-explored issue - namely that the correlation between which experts are used at different layers is weak, with different layers making seemly arbitrary, independent decisions about which experts to use. We hypothesize that this would lead to models that do not specialize to data very strongly.

— Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition

“this interferes with the model's training objective and degrades accuracy”

— GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems

“though they typically rely on a small expert pool (16 to a few hundred) that restricts specialization”

— $\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts

“Compared to existing state-of-the-art MoE baselines (Switch Transformer, MoLE, HydraLoRA), HiLoMoE consistently shows superior efficiency and effectiveness.”

— Hierarchical LoRA MoE for Efficient CTR Model Scaling

“it can suffer from imbalanced expert utilization.”

— Neural Inhibition Improves Dynamic Routing and Mixture of Experts

Beaten on benchmarks

Head-to-head results where a newer method reports beating Switch Transformer. Values are copied from the source paper's tables — verify against the cited paper.

ConceptM³oE beats Switch Transformer
0.600 vs 0.200
F1 · [PBT (4-class) Per-class F1]
ConceptM$^3$oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology
Omni-router beats Switch Transformer
3.9 vs 8.6
test-clean · [8 experts, 559M model]
Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
MoE-X beats Switch Transformer
0.840 vs 0.734
Reconstruction · [Mixture-of-Experts]
Mixture of Experts Made Intrinsically Interpretable
I²MoE-MulT beats Switch Transformer
52.38 vs 47.33
Macro F1 · [IMDB]
I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
∞-MoE beats Switch Transformer
39.8 vs 38.2
Average accuracy · [GPT-2 Small]
$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
Tanbr beats Switch Transformer
75.43 vs 74.41
Avg · [T5-based MoE]
Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inference
HiLoMoE beats Switch Transformer
0.1932 vs 0.1935
LogLoss · [BST + TaobaoAd]
Hierarchical LoRA MoE for Efficient CTR Model Scaling

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.