Fixed Random Classifier Rearrangement for Continual Learning
This addresses the problem of neural networks forgetting old tasks when learning new ones in continual learning, representing an incremental improvement by focusing on classifier constraints.
The paper tackles catastrophic forgetting in continual learning by analyzing how classifier norms affect forgetting and proposing a two-stage algorithm (FRCR) that replaces learnable classifiers with fixed random ones and rearranges new classifier entries. Experimental results show FRCR significantly mitigates forgetting on multiple datasets.
With the explosive growth of data, continual learning capability is increasingly important for neural networks. Due to catastrophic forgetting, neural networks inevitably forget the knowledge of old tasks after learning new ones. In visual classification scenario, a common practice of alleviating the forgetting is to constrain the backbone. However, the impact of classifiers is underestimated. In this paper, we analyze the variation of model predictions in sequential binary classification tasks and find that the norm of the equivalent one-class classifiers significantly affects the forgetting level. Based on this conclusion, we propose a two-stage continual learning algorithm named Fixed Random Classifier Rearrangement (FRCR). In first stage, FRCR replaces the learnable classifiers with fixed random classifiers, constraining the norm of the equivalent one-class classifiers without affecting the performance of the network. In second stage, FRCR rearranges the entries of new classifiers to implicitly reduce the drift of old latent representations. The experimental results on multiple datasets show that FRCR significantly mitigates the model forgetting; subsequent experimental analyses further validate the effectiveness of the algorithm.