Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay
This work addresses the problem of forgetting old classes when learning new ones incrementally in machine learning, but it is incremental as it builds on existing state-of-the-art methods.
The paper tackles catastrophic forgetting in class-incremental learning by proposing data-free generative replay to generate synthetic samples without training data and dual-teacher information distillation for knowledge transfer, achieving performance improvements on CIFAR-100 and ImageNet datasets.
This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our proposed DF-GR, we train a generative model from scratch without using any training data, based on the pre-trained classification model from the past, so we curtail the cost of sharing pre-trained generative models. Second, we introduce dual-teacher information distillation (DT-ID) for knowledge distillation from two teachers to one student. In CIL, we use DT-ID to learn new classes incrementally based on the pre-trained model for old classes and another model (pre-)trained on the new data for new classes. We implemented the proposed schemes on top of one of the state-of-the-art CIL methods and showed the performance improvement on CIFAR-100 and ImageNet datasets.