GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs
This addresses the problem of maintaining general functionality while removing privacy and copyright-related responses in LLMs, representing an incremental improvement.
The paper tackles the trade-off between unlearning and retention in large language models by proposing Gradient Rectified Unlearning (GRU), which regulates gradient updates to minimize side effects, demonstrating practical effectiveness across benchmarks.
Large language model (LLM) unlearning has demonstrated its essential role in removing privacy and copyright-related responses, crucial for their legal and safe applications. However, the pursuit of complete unlearning often comes with substantial costs due to its compromises in their general functionality, leading to a notorious trade-off between unlearning and retention. It motivates this paper to explore enhanced unlearning schemes that can mitigate this trade-off. Specifically, we propose Gradient Rectified Unlearning (GRU), an improved framework that regulates the directions of gradient updates during the unlearning procedure such that their side impacts on other, unrelated responses can be minimized. GRU is easy and general to implement, demonstrating practical effectiveness across a variety of well-established unlearning benchmarks.