Dual-Space Smoothness for Robust and Balanced LLM Unlearning
This addresses privacy, copyright, and safety concerns in LLMs by enhancing unlearning robustness and balance, though it is incremental as it builds on existing unlearning methods.
The paper tackles the problem of catastrophic forgetting and metric imbalance in machine unlearning for large language models, proposing PRISM, a framework that enforces dual-space smoothness to improve robustness and balance, and shows it outperforms state-of-the-art baselines under multiple attacks.
With the rapid advancement of large language models, Machine Unlearning has emerged to address growing concerns around user privacy, copyright infringement, and overall safety. Yet state-of-the-art (SOTA) unlearning methods often suffer from catastrophic forgetting and metric imbalance, for example by over-optimizing one objective (e.g., unlearning effectiveness, utility preservation, or privacy protection) at the expense of others. In addition, small perturbations in the representation or parameter space can be exploited by relearn and jailbreak attacks. To address these challenges, we propose PRISM, a unified framework that enforces dual-space smoothness in representation and parameter spaces to improve robustness and balance unlearning metrics. PRISM consists of two smoothness optimization stages: (i) a representation space stage that employs a robustly trained probe to defend against jailbreak attacks, and (ii) a parameter-space stage that decouples retain-forget gradient conflicts, reduces imbalance, and smooths the parameter space to mitigate relearning attacks. Extensive experiments on WMDP and MUSE, across conversational-dialogue and continuous-text settings, show that PRISM outperforms SOTA baselines under multiple attacks while achieving a better balance among key metrics.