Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
For machine learning practitioners, this highlights a previously overlooked vulnerability in unlearning procedures for second-order optimizers.
The paper shows that second-order optimizers retain residual information after machine unlearning, which is not detectable by first-order analysis, and that controlled state perturbation is needed to erase this geometric memory.
We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn't detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased.