Unlearning via Sparse Representations
This addresses the costly and infeasible nature of existing unlearning techniques for machine learning models, though it appears incremental as it builds on prior work like SCRUB.
The paper tackles the problem of machine unlearning, which erases knowledge about a forget set from a trained model, by proposing a nearly compute-free zero-shot technique based on a discrete representational bottleneck. The result shows that this technique efficiently unlearns the forget set with negligible damage to model performance, performing as well as or better than the state-of-the-art SCRUB method across CIFAR-10, CIFAR-100, and LACUNA-100 datasets while incurring almost no computational cost.
Machine \emph{unlearning}, which involves erasing knowledge about a \emph{forget set} from a trained model, can prove to be costly and infeasible by existing techniques. We propose a nearly compute-free zero-shot unlearning technique based on a discrete representational bottleneck. We show that the proposed technique efficiently unlearns the forget set and incurs negligible damage to the model's performance on the rest of the data set. We evaluate the proposed technique on the problem of \textit{class unlearning} using three datasets: CIFAR-10, CIFAR-100, and LACUNA-100. We compare the proposed technique to SCRUB, a state-of-the-art approach which uses knowledge distillation for unlearning. Across all three datasets, the proposed technique performs as well as, if not better than SCRUB while incurring almost no computational cost.