LLM Unlearning using Gradient Ratio-Based Influence Estimation and Noise Injection
This work addresses the need for effective unlearning in LLMs to comply with legal and ethical requirements, though it is incremental as it builds on existing empirical methods.
The paper tackles the problem of machine unlearning in large language models (LLMs) by proposing GRIN, a framework that uses gradient-ratio-based influence estimation and selective noise injection to improve targeted forgetting of sensitive data, achieving enhanced unlearning performance while preserving model utility on benchmarks like TOFU, WMDP, and SafePKU.
The growing legal and ethical scrutiny of large language models (LLMs) necessitates effective machine unlearning, particularly for sensitive or unauthorized data. Existing empirical methods often yield incomplete forgetting or unintended degradation of unrelated knowledge due to poor localization. In this work, we propose GRIN: a modular and targeted framework for LLM unlearning. GRIN introduces a novel gradient-ratio-based metric to identify parameters most responsible for memorizing forget data. We then perform selective noise injection into these parameters prior to fine-tuning, which improves unlearning performance while maintaining model utility. Finally, we propose new evaluation metrics tailored to the LLM setting and validate our approach on standard benchmarks such as TOFU, WMDP, and SafePKU.