Selective Forgetting of Deep Networks at a Finer Level than Samples
This work is significant for practitioners and researchers dealing with continual learning, data privacy, and robust model deployment, by enabling more granular control over what information deep networks retain or forget.
This paper addresses selective forgetting in deep neural networks at a finer granularity than individual samples, specifically for classification tasks. The authors formulate this as an optimization problem with forgetting, correction, and remembering terms, demonstrating that their methods can make models forget specific information for classification. In some cases, their approach unexpectedly improved model accuracy on datasets containing information to be forgotten but unavailable during the forgetting process.
Selective forgetting or removing information from deep neural networks (DNNs) is essential for continual learning and is challenging in controlling the DNNs. Such forgetting is crucial also in a practical sense since the deployed DNNs may be trained on the data with outliers, poisoned by attackers, or with leaked/sensitive information. In this paper, we formulate selective forgetting for classification tasks at a finer level than the samples' level. We specify the finer level based on four datasets distinguished by two conditions: whether they contain information to be forgotten and whether they are available for the forgetting procedure. Additionally, we reveal the need for such formulation with the datasets by showing concrete and practical situations. Moreover, we introduce the forgetting procedure as an optimization problem on three criteria; the forgetting, the correction, and the remembering term. Experimental results show that the proposed methods can make the model forget to use specific information for classification. Notably, in specific cases, our methods improved the model's accuracy on the datasets, which contains information to be forgotten but is unavailable in the forgetting procedure. Such data are unexpectedly found and misclassified in actual situations.