Mixture-of-experts routing

Most-superseded baselines

Ranked by how many distinct papers critique or beat each method. These are the standard baselines newer work routinely measures against.

1. 1

   MC-SMoE· MC-SMoE

   8 papers critique it · 12 beat it on benchmarks

2. 2

   Switch Transformer· Switch Transformer

   Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

   5 papers critique it · 7 beat it on benchmarks

3. 3

   NAEE· MC-SMoE

   5 papers critique it · 6 beat it on benchmarks

4. 4

   HydraLoRA· HydraLoRA

   HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

   1 papers critique it · 8 beat it on benchmarks

5. 5

   Fiddler· MC-SMoE

   Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

   7 papers critique it · 2 beat it on benchmarks

6. 6

   LoRAMoE· HydraLoRA

   LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

   3 papers critique it · 5 beat it on benchmarks

7. 7

   BTX· BTX

   2 papers critique it · 5 beat it on benchmarks

8. 8

   HC-SMoE· MC-SMoE

   3 papers critique it · 4 beat it on benchmarks

9. 9

   ReMoE· ReMoE

   ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

   3 papers critique it · 4 beat it on benchmarks

10. 10

    Soft MoE· ReMoE

    From Sparse to Soft Mixtures of Experts

    3 papers critique it · 4 beat it on benchmarks

11. 11

    Mixtral-Offloading· MC-SMoE

    5 papers critique it · 2 beat it on benchmarks

12. 12

    Tutel· Switch Transformer

    Tutel: Adaptive Mixture-of-Experts at Scale

    6 papers critique it · 1 beat it on benchmarks