Scholar Feed

Parameter-efficient fine-tuning (LoRA family)

MoELoRA

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

Superseded baseline#8 of 1,113 most-superseded · first seen Feb 20, 2024

Superseded — cited as a baseline and beaten by newer methods

11 papers critique it · 10 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites MoELoRA as a baseline.

Compared to the base model, the three multi-task LoRA method (MOELoRA, MTL-LoRA, and HydraLoRA) are effective, but fail to effectively learn the instruction patterns in the pre-trained weights due to the random initialization of their experts
— CoLA: Collaborative Low-Rank Adaptation
“However, our experimental results indicate that these models are less effective in multi-modal fusion.”
— VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition
“Approaches like MultiLoRA and MoELoRA improve LoRA's multi-task performance in joint training scenarios by integrating multiple LoRAs or utilizing expert routing. However, they fail to strike a good balance between task-specific information and task-information sharing, resulting in suboptimal performance.”
— MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
“many MoE-LoRA variants operate at relatively coarse granularity (e.g., selecting experts at layer/module level), and routing can suffer from imbalance or collapse without careful regularization”
— Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition
“the reliance on routers prevents the adapted parameters from being merged back into the base model, leading to considerable inference overhead and extra storage requirements, thereby hindering real-world deployment”
— ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation
“Compared with LoRA Fine-Tune, MoELoRA has superior anti-forgetting performance due to the multi-experts mixture mechanism, while it fails in some tasks.”
— Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
“they implicitly assume that experts operate independently. In practice, this independence amplifies routing noise, induces sharp and low-entropy gating distributions, and causes the routing mass to concentrate on a small subset of experts”
— TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models
“However, while these methods effectively mitigate interference, they allocate separate LoRA modules per expert, leading to a multiplicative increase in parameter count as the number of experts grows.”
— Less is More: Resource-Efficient Low-Rank Adaptation
“LoRA-MoE lacks fine-grained rank control due to its expert-level gating”
— Adaptive Capacity Allocation for Vision Language Action Fine-tuning
“Although the MoELoRA moelora and HydraLoRA hydralora architectures use different weights for different tokens, they do not adequately address the limitations of shared input-output projections.”
— Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
“the allocation of expert numbers in the LoRA-MoE architecture still relies on manual settings, potentially leading to significant parameter redundancy and overfitting issues, thereby weakening the model's generalization capability and downstream task performance.”
— A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning

Beaten on benchmarks

Head-to-head results where a newer method reports beating MoELoRA. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.