Auditing of Unlearning Algorithms
It provides a practical tool for empirically falsifying unlearning claims, addressing the open challenge of evaluating whether unlearning algorithms truly remove training data influence.
The paper proposes a practical auditor that computes data-dependent lower bounds on the unlearning parameter ε using membership inference attacks, and finds that rigorous unlearning algorithms achieve small ε bounds while empirical methods exhibit large bounds, indicating poor unlearning.
Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes data-dependent lower bounds on the unlearning parameter $\varepsilon$ using membership inference attacks. Evaluating multiple unlearning algorithms, we find a sharp separation: algorithms with rigorous guarantees, such as model clipping and rewind-to-delete, achieve very small $\varepsilon$ bounds that do not falsify their unlearning guarantees, whereas empirical methods such as Hessian-based unlearning, interleaved ascent-descent, ascent on the forget set, and fine-tuning on the retain set exhibit large bounds, indicating poor unlearning. Our auditor provides a practical tool for empirically falsifying unlearning claims through a hypothesis-testing framework, and we validate it on CIFAR-100 and Shakespeare text.