Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
This addresses the safety issue of removing private or harmful data from LLMs, but it is incremental as it builds on existing unlearning approaches with a novel objective and evaluation.
The paper tackles the problem of unlearning specific information in large language models by introducing JensUn, a method using Jensen-Shannon Divergence, which achieves better forget-utility trade-offs and resilience to relearning compared to existing methods, with improvements demonstrated through a new evaluation framework.
Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private data or harmful knowledge acquired during pre-training. However, existing unlearning methods often fall short when subjected to thorough evaluation. To overcome this, we introduce JensUn, where we leverage the Jensen-Shannon Divergence as the training objective for both forget and retain sets for more stable and effective unlearning dynamics compared to commonly used loss functions. In extensive experiments, JensUn achieves better forget-utility trade-off than competing methods, and even demonstrates strong resilience to benign relearning. Additionally, for a precise unlearning evaluation, we introduce LKF, a curated dataset of lesser-known facts that provides a realistic unlearning scenario. Finally, to comprehensively test unlearning methods, we propose (i) employing an LLM as semantic judge instead of the standard ROUGE score, and (ii) using worst-case unlearning evaluation over various paraphrases and input formats. Our improved evaluation framework reveals that many existing methods are less effective than previously thought.