Method Drift›Mixture-of-experts routing
LLaMA-MoE
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingMixture-of-experts routing · first seen Jun 24, 2024
superseded — cited as a baseline and beaten by newer methods
2 papers critique it · 1 beat it on benchmarks
What papers say
Verbatim critique sentences, each from a paper that cites LLaMA-MoE as a baseline.
“require extensive continual training (200B and 7B tokens, respectively)”
— CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference“Unlike existing methods cmoe, llamamoe, llamamoev2 which freeze the assignment and then train the router, DOT-MoE allows the router and the expert assignment to co-adapt.”
— DOT-MoE: Differentiable Optimal Transport for MoEfication
Beaten on benchmarks
Head-to-head results where a newer method reports beating LLaMA-MoE. Values are copied from the source paper's tables — verify against the cited paper.
- CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
CMoE beats LLaMA-MoE · PIQA [Llama-2 7B]
74.34 vs 49.35
- CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
CMoE beats LLaMA-MoE · ARC-C [Llama-2 70B]
49.86 vs 32.40
- CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
CMoE beats LLaMA-MoE · PIQA [Qwen-2.5-7B]
75.93 vs 49.63
- CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
CMoE beats LLaMA-MoE · PIQA [Qwen-3-30B-A3B]
80.23 vs 52.18
- CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
CMoE beats LLaMA-MoE · MMLU [Llama-2 7B 25% sparsity]
44.02 vs 35.09
Newer alternatives
Recent methods in the same sub-problem, not yet superseded in the knowledge base.
- Jun 1, 2026
- May 18, 2026
- Feb 17, 2026
- Mixture-of-Experts (MoE) AdaptationUnderstanding and Harnessing Sparsity in Unified Multimodal ModelsDec 2, 2025
- Elastic Mixture-of-Experts (EMoE)Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-ExpertsSep 26, 2025