GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping
This addresses safety, privacy, and intellectual property concerns in LLMs by enabling effective unlearning of relational and reasoned knowledge, which is a significant advancement over existing flat sentence-level methods.
The paper tackles the problem of unlearning structured knowledge in large language models by introducing a benchmark for knowledge graph facts and a novel framework that leverages graph connectivity to precisely remove target facts while preserving related knowledge, achieving an unlearning efficacy of 1.000 and locality of 0.839 on LLaMA-3-8B and Mistral-7B.
Unlearning knowledge is a pressing and challenging task in Large Language Models (LLMs) because of their unprecedented capability to memorize and digest training data at scale, raising more significant issues regarding safety, privacy, and intellectual property. However, existing works, including parameter editing, fine-tuning, and distillation-based methods, are all focused on flat sentence-level data but overlook the relational, multi-hop, and reasoned knowledge in naturally structured data. In response to this gap, this paper introduces Graph Oblivion and Node Erasure (GONE), a benchmark for evaluating knowledge unlearning over structured knowledge graph (KG) facts in LLMs. This KG-based benchmark enables the disentanglement of three effects of unlearning: direct fact removal, reasoning-based leakage, and catastrophic forgetting. In addition, Neighborhood-Expanded Distribution Shaping (NEDS), a novel unlearning framework, is designed to leverage graph connectivity and identify anchor correlated neighbors, enforcing a precise decision boundary between the forgotten fact and its semantic neighborhood. Evaluations on LLaMA-3-8B and Mistral-7B across multiple knowledge editing and unlearning methods showcase NEDS's superior performance (1.000 on unlearning efficacy and 0.839 on locality) on GONE and other benchmarks. Code is available at https://anonymous.4open.science/r/GONE-4679/.