Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
This addresses the challenge of effectively removing sensitive or copyrighted data from LLMs without degrading performance, representing a novel method for a known bottleneck in unlearning.
The paper tackles the problem of catastrophic collapse in large language model unlearning by proposing Negative Preference Optimization (NPO), which achieves better balance between unlearning undesirable data and preserving model utilities, notably enabling reasonable unlearning of 50% or more of training data on the TOFU benchmark where existing methods struggle with 10%.
Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from catastrophic collapse -- a drastic degradation of the model's utilities. In this paper, we propose Negative Preference Optimization (NPO), a simple alignment-inspired method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catastrophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve a better balance between unlearning the undesirable data and maintaining the model's utilities. We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish. Remarkably, on TOFU, NPO-based methods are the first to achieve reasonable unlearning results in forgetting 50% (or more) of the training data, whereas existing methods already struggle with forgetting 10% of training data.