Is DeepSeekMoE superseded?

DeepSeekMoE (Mixture-of-experts routing): superseded — cited as a baseline and beaten by newer methods. 0 paper(s) critique it, 6 beat it on benchmarks — #13 of 1370 most-superseded. Sub-problem: cluster led by ReMoE. Newer alternatives in the same sub-problem include ProbMoE, Grouter, DECO, AIR-MoE, SPHERE.

Method Drift›Mixture-of-experts routing

Superseded baseline#13 of 1,370 most-superseded

DeepSeekMoE

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Mixture-of-experts routing · first seen Jan 11, 2024

superseded — cited as a baseline and beaten by newer methods

0 papers critique it · 6 beat it on benchmarks

Beaten on benchmarks

Head-to-head results where a newer method reports beating DeepSeekMoE. Values are copied from the source paper's tables — verify against the cited paper.

IDA-MoE beats DeepSeekMoE · MMVet [StableLM-1.6B + CLIP-336]
29.5 vs 28.7
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
IDA-MoE beats DeepSeekMoE · MME-per [StableLM-1.6B + CLIP-336]
1372.0 vs 1354.5
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
IDA-MoE beats DeepSeekMoE · PoPE [StableLM-1.6B + CLIP-336]
86.8 vs 85.6
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
IDA-MoE beats DeepSeekMoE · SQA^I [StableLM-1.6B + CLIP-336]
65.1 vs 61.2
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
IDA-MoE beats DeepSeekMoE · TextVQA [StableLM-1.6B + CLIP-336]
51.9 vs 49.2
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
IDA-MoE beats DeepSeekMoE · VizWiz [StableLM-1.6B + CLIP-336]
43.1 vs 38.4
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
IDA-MoE beats DeepSeekMoE · MMB [StableLM-1.6B + CLIP-336]
63.2 vs 60.4
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
DS-MoE-6B beats DeepSeekMoE · TPS [H100-80GB GPU]
4603.9 vs 3144.1
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
Mistral-MoCE beats DeepSeekMoE · Avg [Cross-model comparison]
55.03 vs 40.77
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
OmniMoE beats DeepSeekMoE · Avg [6.4B-A1.7B models]
50.9 vs 50.2
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
BlockFFN beats DeepSeekMoE · CLS_8 [Small]
71.38 vs 49.27
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
BlockFFN beats DeepSeekMoE · CLS_8 [Medium]
75.87 vs 66.22
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

Newer alternatives

Recent methods in the same sub-problem, not yet superseded in the knowledge base.