Is MoELoRA superseded?

MoELoRA (Mixture-of-experts routing): superseded — cited as a baseline and beaten by newer methods. 1 paper(s) critique it, 5 beat it on benchmarks — #17 of 1370 most-superseded. Sub-problem: cluster led by HydraLoRA. Newer alternatives in the same sub-problem include PARAMΔ Integration into Upcycled MoE, MEMIT-like framework for MoE, HELLoRA, SDG-MoE, Marco-MoE.

Mixture-of-experts routing

MoELoRA

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

Superseded baseline#17 of 1,370 most-superseded · first seen Feb 20, 2024

Superseded — cited as a baseline and beaten by newer methods

1 papers critique it · 5 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites MoELoRA as a baseline.

These methods still exhibit substantial performance degradation on earlier tasks as training progresses

— SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

Beaten on benchmarks

Head-to-head results where a newer method reports beating MoELoRA. Values are copied from the source paper's tables — verify against the cited paper.

LLaVA-CMoE beats MoELoRA
59.23 vs 44.24
Mean · [Last]
LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
SAME (MOMO) beats MoELoRA
66.82 vs 50.58
Accuracy · [all tasks CoIN]
SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
PASs-MoE beats MoELoRA
43.22 vs 35.89
Acc · [Arts VQA]
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
LiMELoRA beats MoELoRA
84.98 vs 84.08
Commonsense Reasoning · [Commonsense Reasoning]
LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.