Is HydraLoRA superseded?

HydraLoRA (Mixture-of-experts routing): heavily superseded — a standard baseline that newer methods routinely beat. 1 paper(s) critique it, 8 beat it on benchmarks — #3 of 1370 most-superseded. Sub-problem: cluster led by HydraLoRA. Newer alternatives in the same sub-problem include PARAMΔ Integration into Upcycled MoE, MEMIT-like framework for MoE, HELLoRA, SDG-MoE, Marco-MoE.

Mixture-of-experts routing

HydraLoRA

HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

Heavily superseded#3 of 1,370 most-superseded · first seen Apr 30, 2024

Heavily superseded — a standard baseline that newer methods routinely beat

1 papers critique it · 8 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites HydraLoRA as a baseline.

Compared to existing state-of-the-art MoE baselines (Switch Transformer, MoLE, HydraLoRA), HiLoMoE consistently shows superior efficiency and effectiveness. On average, it improves AUC by 0.08% and reduces LogLoss by 0.10% compared to the best competing MoE variant (HydraLoRA). At the same time, HiLoMoE reduces parameter count by an average of 4.04K, which is equivalent to a 21.0% reduction relative to the most parameter-efficient MoE competitor (HydraLoRA).

— Hierarchical LoRA MoE for Efficient CTR Model Scaling

Beaten on benchmarks

Head-to-head results where a newer method reports beating HydraLoRA. Values are copied from the source paper's tables — verify against the cited paper.

SMoRA beats HydraLoRA
36.66 vs 32.41
Average · [Multi-Domain]
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
LoPE beats HydraLoRA
40.78 vs 37.96
MMLU · [30% noise ratio]
Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
FourierMoE beats HydraLoRA
73.24 vs 69.63
AVG. · [LLaMA-3 8B]
FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
GOAT beats HydraLoRA
60.20 vs 57.39
GSM8K · [Natural Language Generation (NLG)]
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
MoORE (L=8) beats HydraLoRA
85.11 vs 83.84
Overall · [CSR-MTL multi-task adaptation]
MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
HiLoMoE beats HydraLoRA
0.4341 vs 0.4374
LogLoss · [DIEN + KuaiVideo]
Hierarchical LoRA MoE for Efficient CTR Model Scaling
LoRA-SMoE-S(60%) beats HydraLoRA
83.2 vs 82.9
average · [multi-task benchmark]
A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
LiMEDoRA beats HydraLoRA
78.12 vs 78.11
Vision Benchmark · [Vision Benchmark]
LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.