Scholar Feed

Parameter-efficient fine-tuning (LoRA family)

O-LoRA

OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models

Superseded baseline#21 of 1,113 most-superseded · first seen Jun 3, 2024

Superseded — cited as a baseline and beaten by newer methods

6 papers critique it · 6 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites O-LoRA as a baseline.

However, this work does not provide an effective way for LoRA composition.
— LoRA-Based Continual Learning with Constraints on Critical Parameter Changes
“While O-LoRA mitigates interference via orthogonal gradient updates, its additive updates ($W + \Delta W$) can distort the intrinsic geometry of LLM parameters”
— Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
“The internal gauge freedom, scale ambiguity, and rank collapse of the $BA^$ structure persist throughout training.”
— OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb
“Our proposed CLoRA imposes orthogonal regularization similar to O-LoRA, but the regularization matrix is not restricted to be the previous learned parameter, thus CLoRA can be used for one-stage continued training whereas O-LoRA not.”
— Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
“In contrast to O-LoRA, we preserve the magnitude of updates through Fisher-based penalties, providing insights that may inform the broader application of regularization techniques within PECL—an area that remains underexplored in the current literature.”
— Revisiting Weight Regularization for Low-Rank Continual Learning
“While both approaches are effective, they differ in managing parameter updates. MoE-based approaches aggregate task-specific LoRA weights via attention mechanisms, whereas orthogonality-based methods regulate LoRA parameter updates by constraining gradient update directions. However, neither method directly examines how parameter shifts evolve across tasks (i.e., the internal dynamics of parameter space), which is a crucial yet underexplored factor in model forgetting.”
— Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces

Beaten on benchmarks

Head-to-head results where a newer method reports beating O-LoRA. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.