SRIL: Selective Regularization for Class-Incremental Learning
This addresses the problem of catastrophic forgetting in deep learning models for incremental learning scenarios, representing an incremental improvement with novel regularization techniques.
The paper tackles catastrophic forgetting in class-incremental learning by proposing a selective regularization method that balances plasticity and stability, achieving superior performance over existing methods on CIFAR-100, ImageNet-Subset, and ImageNet-Full datasets.
Human intelligence gradually accepts new information and accumulates knowledge throughout the lifespan. However, deep learning models suffer from a catastrophic forgetting phenomenon, where they forget previous knowledge when acquiring new information. Class-Incremental Learning aims to create an integrated model that balances plasticity and stability to overcome this challenge. In this paper, we propose a selective regularization method that accepts new knowledge while maintaining previous knowledge. We first introduce an asymmetric feature distillation method for old and new classes inspired by cognitive science, using the gradient of classification and knowledge distillation losses to determine whether to perform pattern completion or pattern separation. We also propose a method to selectively interpolate the weight of the previous model for a balance between stability and plasticity, and we adjust whether to transfer through model confidence to ensure the performance of the previous class and enable exploratory learning. We validate the effectiveness of the proposed method, which surpasses the performance of existing methods through extensive experimental protocols using CIFAR-100, ImageNet-Subset, and ImageNet-Full.