Investigating Model Editing for Unlearning in Large Language Models
This addresses the challenge of efficiently unlearning specific information in large language models, though it appears incremental as it adapts existing editing techniques rather than introducing fundamentally new approaches.
The paper tackles the problem of removing unwanted information from large language models by adapting model editing algorithms (ROME, IKE, WISE) for unlearning, showing they can exceed baseline unlearning methods in forgetting quality depending on the setting, but still struggle to fully remove information without damaging overall model performance.
Machine unlearning aims to remove unwanted information from a model, but many methods are inefficient for LLMs with large numbers of parameters or fail to fully remove the intended information without degrading performance on knowledge that should be retained. Model editing algorithms solve a similar problem of changing information in models, but they focus on redirecting inputs to a new target rather than removing that information altogether. In this work, we explore the editing algorithms ROME, IKE, and WISE and design new editing targets for an unlearning setting. Through this investigation, we show that model editing approaches can exceed baseline unlearning methods in terms of quality of forgetting depending on the setting. Like traditional unlearning techniques, they struggle to encapsulate the scope of what is to be unlearned without damage to the overall model performance.