UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets
This addresses the issue of effectively removing harmful information from LLMs for safer deployment, representing an incremental improvement over existing unlearning methods.
The paper tackles the problem of LLMs retaining harmful information by showing that existing unlearning methods fail because models can reconstruct target content through logically related knowledge, and proposes UIPE to remove correlated knowledge, achieving significant performance improvements on the TOFU benchmark.
Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of various mainstream LLM unlearning methods on the TOFU benchmark.