Orthogonal Subspace Learning for Language Model Continual Learning
This addresses the problem of performance degradation in language models for continual learning scenarios, offering an incremental improvement over existing methods.
The paper tackles catastrophic forgetting in large language models during sequential task learning by proposing orthogonal low-rank adaptation (O-LoRA), which learns tasks in orthogonal low-rank subspaces to minimize interference, resulting in outperforming state-of-the-art methods on benchmarks and better preserving generalization on unseen tasks.
Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.