Mixture-of-experts routing

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

5 critique · 7 beaten on benchmarks

8 critique · 12 beaten on benchmarks

HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

1 critique · 8 beaten on benchmarks

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

0 critique · 6 beaten on benchmarks

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

2 critique · 2 beaten on benchmarks

From Sparse to Soft Mixtures of Experts

3 critique · 4 beaten on benchmarks

Tutel: Adaptive Mixture-of-Experts at Scale

6 critique · 1 beaten on benchmarks

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

7 critique · 2 beaten on benchmarks

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

4 critique · 1 beaten on benchmarks

Mixtral of Experts

1 critique · 2 beaten on benchmarks

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

3 critique · 5 beaten on benchmarks

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

3 critique · 2 beaten on benchmarks