SIMU: Selective Influence Machine Unlearning
This addresses the need for safe and effective unlearning in LLMs, though it is incremental as it builds on existing second-order optimizer-based methods.
The paper tackles the problem of machine unlearning in Large Language Models, where existing methods compromise model utility when forgetting sensitive information, and proposes SIMU, a two-step framework that selectively updates critical neurons to achieve comparable unlearning while better retaining original knowledge.
The undesired memorization of sensitive information by Large Language Models (LLMs) has emphasized the need for safety mechanisms that can regulate model behavior. This has led to the development of machine unlearning techniques that enable models to precisely forget sensitive and unwanted information. For machine unlearning, first-order and second-order optimizer-based methods have shown significant progress in enabling LLMs to forget targeted information. However, in doing so, these approaches often compromise the model's original capabilities, resulting in unlearned models that struggle to retain their prior knowledge and overall utility. To address this, we propose Selective Influence Machine Unlearning (SIMU), a two-step framework that enhances second-order optimizer-based unlearning by selectively updating only the critical neurons responsible for encoding the forget-set. By constraining updates to these targeted neurons, SIMU achieves comparable unlearning efficacy while substantially outperforming current methods in retaining the model's original knowledge.