Lory (Mixture-of-experts routing): superseded — cited as a baseline and beaten by newer methods. 3 paper(s) critique it, 1 beat it on benchmarks — #34 of 1370 most-superseded. Sub-problem: cluster led by DeepSeekMoE. Newer alternatives in the same sub-problem include ProbMoE, Grouter, DECO, SPHERE, AIR-MoE.

Mixture-of-experts routing

Lory

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

Superseded baseline#34 of 1,370 most-superseded · first seen May 6, 2024

Superseded — cited as a baseline and beaten by newer methods

3 papers critique it · 1 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Lory as a baseline.

But it underperforms vanilla MoE with TopK routing.

— ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

“However, these approaches still require the model to construct its load-balanced structure on-the-fly during training.”

— Grouter: Decoupling Routing from Representation for Accelerated MoE Training

“While effective in sequence-based or semantic-based routing scenarios, the computational cost of these operations renders them unsuitable for token-level routing, where efficiency is critical.”

— Efficiently Editing Mixture-of-Experts Models with Compressed Experts

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.