SAGE: Retain-Aware Post-Hoc Sanitization of Final Unlearning Vector
For practitioners of LLM unlearning, SAGE offers a lightweight, complementary fix to improve retention in any existing unlearning method.
LLM unlearning methods suffer from a trade-off between forgetting undesirable knowledge and retaining other capabilities. The authors propose SAGE, a post-hoc method that sanitizes the final update vector to reduce retention damage without rerunning unlearning, consistently improving the retain-forget trade-off across multiple methods and benchmarks.
Large Language Model (LLM) unlearning aims to remove undesirable knowledge or behaviors while preserving retained capabilities. Current unlearning methods all involve a trade-off between unlearning and retention. We have found that the retention activation bias can also be used to quantify the damage an unlearning method inflicts on retention, without considering the specific implementation of the unlearning process. This allows us to restore retention performance for any unlearning method using a post-hoc approach. Therefore, we propose a complementary post-hoc setting to sanitize the final update vector without rerunning the original unlearning pipeline. In this setting, we design SAGE, Spectral Activation-GEometry Sanitization, a source-agnostic correction for final unlearning updates. SAGE collects real module inputs from a small retain proxy, extracts their dominant activation geometry, and solves a source-anchored optimization objective in closed form, which suppresses update components aligned with high-energy retained directions while preserving the source method's forgetting carrier. Across multiple unlearning methods, model scales, and benchmarks, SAGE consistently relieves the retain-forget trade-off, identifying post-hoc sanitization of final vectors as a practical and underexplored axis for machine unlearning.