On the Soft-Subnetwork for Few-shot Class Incremental Learning
This addresses the challenge of catastrophic forgetting and overfitting in incremental learning with few samples, which is crucial for real-world applications like robotics or personalization, though it is an incremental improvement over existing methods.
The paper tackles the problem of few-shot class incremental learning (FSCIL), where models must learn new classes with limited data while retaining knowledge from previous ones, and proposes Soft-SubNetworks (SoftNet) to achieve this by jointly learning weights and adaptive soft masks, resulting in performance that surpasses state-of-the-art baselines on benchmark datasets.
Inspired by Regularized Lottery Ticket Hypothesis (RLTH), which hypothesizes that there exist smooth (non-binary) subnetworks within a dense network that achieve the competitive performance of the dense network, we propose a few-shot class incremental learning (FSCIL) method referred to as \emph{Soft-SubNetworks (SoftNet)}. Our objective is to learn a sequence of sessions incrementally, where each session only includes a few training instances per class while preserving the knowledge of the previously learned ones. SoftNet jointly learns the model weights and adaptive non-binary soft masks at a base training session in which each mask consists of the major and minor subnetwork; the former aims to minimize catastrophic forgetting during training, and the latter aims to avoid overfitting to a few samples in each new training session. We provide comprehensive empirical validations demonstrating that our SoftNet effectively tackles the few-shot incremental learning problem by surpassing the performance of state-of-the-art baselines over benchmark datasets.