Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning
This addresses privacy risks in language models by providing a robust unlearning method, though it is incremental as it builds on existing unlearning techniques.
The paper tackles the problem of spurious unlearning neurons in large language models that cause shallow alignment and vulnerability to relearning, and introduces Ssiuu, a method that reliably erases target knowledge and outperforms baselines in adversarial and benign retraining scenarios.
Large language models trained on web-scale data can memorize private or sensitive knowledge, raising significant privacy risks. Although some unlearning methods mitigate these risks, they remain vulnerable to "relearning" during subsequent training, allowing a substantial portion of forgotten knowledge to resurface. In this paper, we show that widely used unlearning methods cause shallow alignment: instead of faithfully erasing target knowledge, they generate spurious unlearning neurons that amplify negative influence to hide it. To overcome this limitation, we introduce Ssiuu, a new class of unlearning methods that employs attribution-guided regularization to prevent spurious negative influence and faithfully remove target knowledge. Experimental results confirm that our method reliably erases target knowledge and outperforms strong baselines across two practical retraining scenarios: (1) adversarial injection of private data, and (2) benign attack using an instruction-following benchmark. Our findings highlight the necessity of robust and faithful unlearning methods for safe deployment of language models.