Scholar Feed

Mixture-of-experts routing

ReMoE

ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

Superseded baseline#14 of 1,370 most-superseded · first seen Dec 19, 2024

Superseded — cited as a baseline and beaten by newer methods

3 papers critique it · 4 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites ReMoE as a baseline.

While this directly tackles the gradient bottleneck, the resulting "soft" routing needs an auxiliary loss to enforce sparsity. In practice, these can inject interference gradients, complicate tuning, and dampen expert specialization~Wang2024LossFreeBalancing.
— DirMoE: Dirichlet-routed Mixture of Experts
“However, these approaches still require the model to construct its load-balanced structure on-the-fly during training.”
— Grouter: Decoupling Routing from Representation for Accelerated MoE Training
“Both DynMoE and ReMoE face the challenge of needing explicit mechanisms to manage the upper bound on the number of activated experts so as to avoid potential high computation overhead.”
— Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

Beaten on benchmarks

Head-to-head results where a newer method reports beating ReMoE. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.