Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization
This addresses the problem of data removal in trained models for applications requiring privacy or compliance, representing a novel method for a known bottleneck.
The paper tackles the challenge of machine unlearning by proposing OrthoGrad, a method that removes problematic training data without compromising model performance, and demonstrates its effectiveness on benchmarks like automatic speech recognition, outperforming competing methods.
Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the unlearn data with the gradient descent on a retain set representing the training data. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.