InfLoRA

“However, as the number of sequential tasks increases, the continual expansion of prompts or LoRA components, along with the growing storage costs of sample features, becomes inefficient and unsustainable.”

“While they correctly identify the need for orthogonality between tasks, they attempt to enforce it at a high level without addressing a fundamental flaw in the LoRA optimization process itself. Specifically, independent Euclidean updates to the low-rank factors ($A$ and $B$) cause their composite effect to deviate from any intended orthogonal path.”

“While both approaches are effective, they differ in managing parameter updates. MoE-based approaches aggregate task-specific LoRA weights via attention mechanisms, whereas orthogonality-based methods regulate LoRA parameter updates by constraining gradient update directions. However, neither method directly examines how parameter shifts evolve across tasks (i.e., the internal dynamics of parameter space), which is a crucial yet underexplored factor in model forgetting.”