Towards Natural Machine Unlearning
This addresses the issue of unnatural and over-forgetting in machine unlearning for AI systems, though it appears incremental as a first step towards more natural methods.
The paper tackles the problem of unnatural machine unlearning, where current methods modify forgetting data with incorrect labels, leading to over-forgetting, and proposes injecting correct information from remaining data to suppress forgotten information, resulting in significant outperformance over state-of-the-art approaches with reduced over-forgetting and strong robustness to hyperparameters.
Machine unlearning (MU) aims to eliminate information that has been learned from specific training data, namely forgetting data, from a pre-trained model. Currently, the mainstream of existing MU methods involves modifying the forgetting data with incorrect labels and subsequently fine-tuning the model. While learning such incorrect information can indeed remove knowledge, the process is quite unnatural as the unlearning process undesirably reinforces the incorrect information and leads to over-forgetting. Towards more \textit{natural} machine unlearning, we inject correct information from the remaining data to the forgetting samples when changing their labels. Through pairing these adjusted samples with their labels, the model will tend to use the injected correct information and naturally suppress the information meant to be forgotten. Albeit straightforward, such a first step towards natural machine unlearning can significantly outperform current state-of-the-art approaches. In particular, our method substantially reduces the over-forgetting and leads to strong robustness to hyperparameters, making it a promising candidate for practical machine unlearning.