Knowledge Swapping via Learning and Unlearning
This work addresses the problem of efficient knowledge management in pretrained models for machine learning practitioners and researchers, providing an incremental yet useful solution.
The authors tackled the problem of selectively regulating knowledge in a pretrained model, enabling the forgetting of specific information while retaining essential knowledge and acquiring new knowledge, with comprehensive experiments validating the effectiveness of their proposed strategy. The results showed the strategy's effectiveness across various tasks like image classification, object detection, and semantic segmentation.
We introduce \textbf{Knowledge Swapping}, a novel task designed to selectively regulate knowledge of a pretrained model by enabling the forgetting of user\-specified information, retaining essential knowledge, and acquiring new knowledge simultaneously. By delving into the analysis of knock-on feature hierarchy, we find that incremental learning typically progresses from low\-level representations to higher\-level semantics, whereas forgetting tends to occur in the opposite direction\-starting from high-level semantics and moving down to low-level features. Building upon this, we propose to benchmark the knowledge swapping task with the strategy of \textit{Learning Before Forgetting}. Comprehensive experiments on various tasks like image classification, object detection, and semantic segmentation validate the effectiveness of the proposed strategy. The source code is available at \href{https://github.com/xingmingyu123456/KnowledgeSwapping}{https://github.com/xingmingyu123456/KnowledgeSwapping}.