RULER: Representation-Level Verification of Machine Unlearning
For practitioners and regulators of machine unlearning, RULER reveals that current output-level verification is insufficient, providing more rigorous metrics to ensure privacy.
RULER introduces representation-level verification metrics for machine unlearning, detecting residual information in intermediate representations that output-level tests miss. Across four approximate unlearning methods, M2 found significant residuals in 10 of 12 conditions (p<0.05), and M4 diagnosed identity memorization in face recognition models.
Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current protocols verify this at the output level through membership inference, retain accuracy, and forget-set accuracy, but a model can satisfy all three whilst still encoding forgotten records in its intermediate representations. We introduce RULER, a set of representation-level verification metrics. The oracle-comparative metric M2 measures whether forget-set records occupy the same representational position as in a model retrained without them. The oracle-free metric M4 detects residuals from the unlearned model's internal similarity structure alone, without retraining. Four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect sizes growing as the forget fraction increases. A fifth method, Bad Teacher, shows the same residuals despite a different forgetting mechanism. M4 acts as a pre-unlearning diagnostic across tabular, image, clinical text, and face-identity settings: it detects identity-level memorisation in face recognition models where no tested method fully erases the signal.