Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs
This addresses privacy and safety issues in LLMs by providing a more stable unlearning method, though it is incremental as it builds on existing gradient difference and LoRA-based techniques.
The paper tackled the problem of unstable and unreliable machine unlearning in large language models by proposing a bounded parameter-efficient approach, which achieved substantial improvements in forgetting while preserving retention across multiple benchmarks and model scales up to 8B parameters.
Machine unlearning in large language models (LLMs) is essential for privacy and safety; however, existing approaches remain unstable and unreliable. A widely used strategy, the gradient difference method, applies gradient descent on retained data while performing gradient ascent on forget data, the data whose influence should be removed. However, when combined with cross-entropy loss, this procedure causes unbounded growth of weights and gradients, leading to training instability and degrading both forgetting and retention. We provide a theoretical framework that explains this failure, explicitly showing how ascent on the forget set destabilizes optimization in the feedforward MLP layers of LLMs. Guided by this insight, we propose Bounded Parameter-Efficient Unlearning, a parameter-efficient approach that stabilizes LoRA-based fine-tuning by applying bounded functions to MLP adapters. This simple modification controls the weight dynamics during ascent, enabling the gradient difference method to converge reliably. Across the TOFU, TDEC, and MUSE benchmarks, and across architectures and scales from 125M to 8B parameters, our method achieves substantial improvements in forgetting while preserving retention, establishing a novel theoretically grounded and practically scalable framework for unlearning in LLMs.