Method Drift›Parameter-efficient fine-tuning (LoRA family)
O-LoRA
OLoRA: Orthonormal Low-Rank Adaptation of Large Language ModelsParameter-efficient fine-tuning (LoRA family) · first seen Jun 3, 2024
superseded — cited as a baseline and beaten by newer methods
6 papers critique it · 6 beat it on benchmarks
What papers say
Verbatim critique sentences, each from a paper that cites O-LoRA as a baseline.
“However, this work does not provide an effective way for LoRA composition.”
— LoRA-Based Continual Learning with Constraints on Critical Parameter Changes“While O-LoRA mitigates interference via orthogonal gradient updates, its additive updates ($W + \Delta W$) can distort the intrinsic geometry of LLM parameters”
— Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models“The internal gauge freedom, scale ambiguity, and rank collapse of the $BA^$ structure persist throughout training.”
— OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb“Our proposed CLoRA imposes orthogonal regularization similar to O-LoRA, but the regularization matrix is not restricted to be the previous learned parameter, thus CLoRA can be used for one-stage continued training whereas O-LoRA not.”
— Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models“In contrast to O-LoRA, we preserve the magnitude of updates through Fisher-based penalties, providing insights that may inform the broader application of regularization techniques within PECL—an area that remains underexplored in the current literature.”
— Revisiting Weight Regularization for Low-Rank Continual Learning“While both approaches are effective, they differ in managing parameter updates. MoE-based approaches aggregate task-specific LoRA weights via attention mechanisms, whereas orthogonality-based methods regulate LoRA parameter updates by constraining gradient update directions. However, neither method directly examines how parameter shifts evolve across tasks (i.e., the internal dynamics of parameter space), which is a crucial yet underexplored factor in model forgetting.”
— Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces
Beaten on benchmarks
Head-to-head results where a newer method reports beating O-LoRA. Values are copied from the source paper's tables — verify against the cited paper.
- Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
OLieRA beats O-LoRA · Average Accuracy [Standard CL Benchmark, T5-large]
79.6 vs 75.8
- Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
OLieRA beats O-LoRA · Average Accuracy [Large Number of Tasks, T5-large]
72.6 vs 69.6
- Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
OLieRA beats O-LoRA · Average Accuracy [LLaMA-7B model]
77.7 vs 76.1
- ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editing
ACE-LoRA beats O-LoRA · Overall Score [Avg. metric]
8.8639 vs 7.7231
- Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
CLoRA beats O-LoRA · avg. [Standard CL Benchmark]
79.0 vs 75.8
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · AP [Order 1, T5-large]
47.84 vs 26.37
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · FT [Order 1, T5-large]
2.26 vs 19.15
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · AP [Order 2, T5-large]
46.84 vs 32.83
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · FT [Order 2, T5-large]
2.91 vs 11.99
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · AP [Order 3, T5-large]
73.37 vs 70.98
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · FT [Order 3, T5-large]
3.02 vs 3.69
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
GainLoRA(O-LoRA) beats O-LoRA · AP [Order 4, T5-large]
76.01 vs 71.21
Newer alternatives
Recent methods in the same sub-problem, not yet superseded in the knowledge base.
- G2LoRAG2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed GraphsJun 1, 2026
- CoDyRATake Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual LearningMay 27, 2026
- May 27, 2026
- May 26, 2026
- Beyond Feature FusionBeyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty EstimationApr 17, 2026
- Sequential Fine-Tuning with LoRASimple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement LearningMar 12, 2026
- MAGE (Mixture and Aggregation of General LoRA and Expert LoRA)Continual-NExT: A Unified Comprehension And Generation Continual Learning FrameworkFeb 20, 2026
- Feb 19, 2026
- PS-LoRA (Parameter Stability LoRA)Resolving Conflicts in Lifelong Learning via Aligning Updates in SubspacesNov 28, 2025