Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
This addresses the problem of task interference in continual learning for LLMs, offering a novel geometric approach that is incremental over prior orthogonality-based methods.
The paper tackles catastrophic forgetting in large language models during sequential multi-task learning by proposing OLieRA, which preserves geometric structure using Lie group theory and orthogonality constraints, achieving state-of-the-art results on the Standard CL benchmark and top performance in large-task settings.
Large language models (LLMs) are prone to catastrophic forgetting in sequential multi-task settings. Parameter regularization methods such as O-LoRA and N-LoRA alleviate task interference by enforcing low-rank subspace orthogonality, but they overlook the fact that conventional additive fine-tuning disrupts the intrinsic geometric structure of LLM parameters, limiting performance. Our key insight is that the parameter space of LLMs possesses a geometric structure, which must be preserved in addition to enforcing orthogonality. Based on this, we propose Orthogonal Low-rank Adaptation in Lie Groups (OLieRA), which introduces Lie group theory into LLM fine-tuning: leveraging multiplicative updates to preserve parameter geometry while applying orthogonality constraints to task subspaces. Experiments demonstrate that OLieRA achieves state-of-the-art results on the Standard CL benchmark and remains among the top-performing methods in the Large Number of Tasks setting.