Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning
This addresses the problem of catastrophic forgetting in incremental learning with limited data for AI systems requiring continuous adaptation.
The paper tackles few-shot class incremental learning (FSCIL) by introducing a semantic-aware knowledge distillation algorithm that uses word embeddings and an attention mechanism to align visual and semantic vectors, establishing new state-of-the-art results on MiniImageNet, CUB200, and CIFAR100 datasets.
Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation process. Furthermore, we propose a method based on an attention mechanism on multiple parallel embeddings of visual data to align visual and semantic vectors, which reduces issues related to catastrophic forgetting. Via experiments on MiniImageNet, CUB200, and CIFAR100 dataset, we establish new state-of-the-art results by outperforming existing approaches.