On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning
This work addresses catastrophic forgetting for practitioners using PEFT methods in continual learning scenarios, offering incremental insights into update subspace design.
The paper tackled the problem of catastrophic forgetting in low-rank decomposition-based parameter-efficient fine-tuning (PEFT) during sequential learning, finding that tensor-based decompositions like LoRETTA mitigate forgetting by capturing richer structural information, while structurally aligned parameterizations like WeGeFT preserve pretrained representations.
Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding catastrophic forgetting -- remains insufficiently understood. In this work, we present an empirical study showing that forgetting is strongly influenced by the geometry and parameterization of the update subspace. While methods that restrict updates to small, shared matrix subspaces often suffer from task interference, tensor-based decompositions (e.g., LoRETTA) mitigate forgetting by capturing richer structural information within ultra-compact budgets, and structurally aligned parameterizations (e.g., WeGeFT) preserve pretrained representations. Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.