Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
For practitioners needing to sequentially delete data from models, this method prevents catastrophic forgetting without dynamic routing.
Continual machine unlearning faces interference when sequentially removing data. The proposed SVD-based orthogonal subspace projection for LoRA maintains baseline accuracy (~58.1%) after 30 unlearning tasks on CIFAR-100, compared to state-of-the-art static fusion dropping to 12.70%.
Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing \textit{strong interference} between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39\% to 12.70\%, whereas the proposed in-training constrained optimization maintains baseline performance ($\sim$58.1\%) while preserving strong unlearning efficacy.