Breaking Chains: Unraveling the Links in Multi-Hop Knowledge Unlearning
This work addresses a vulnerability in unlearning methods for LLMs, which is important for privacy and copyright protection, but it is incremental as it builds on existing techniques.
The paper tackled the problem of existing unlearning techniques failing to completely remove multi-hop knowledge in large language models when intermediate hops are unlearned, and proposed MUNCH, an uncertainty-based approach that effectively addresses this issue by breaking down queries and leveraging model uncertainty.
Large language models (LLMs) serve as giant information stores, often including personal or copyrighted data, and retraining them from scratch is not a viable option. This has led to the development of various fast, approximate unlearning techniques to selectively remove knowledge from LLMs. Prior research has largely focused on minimizing the probabilities of specific token sequences by reversing the language modeling objective. However, these methods still leave LLMs vulnerable to adversarial attacks that exploit indirect references. In this work, we examine the limitations of current unlearning techniques in effectively erasing a particular type of indirect prompt: multi-hop queries. Our findings reveal that existing methods fail to completely remove multi-hop knowledge when one of the intermediate hops is unlearned. To address this issue, we propose MUNCH, a simple uncertainty-based approach that breaks down multi-hop queries into subquestions and leverages the uncertainty of the unlearned model in final decision-making. Empirical results demonstrate the effectiveness of our framework, and MUNCH can be easily integrated with existing unlearning techniques, making it a flexible and useful solution for enhancing unlearning processes.