Prototype-based Incremental Few-Shot Semantic Segmentation
This work addresses the problem of efficiently adapting semantic segmentation models to new classes with limited data for practitioners, offering an incremental improvement to existing methods.
This paper introduces Incremental Few-Shot Segmentation (iFSS), a new task to extend pretrained segmentation models with new classes using few annotated images and without old training data. The proposed method, Prototype-based Incremental Few-Shot Segmentation (PIFS), outperforms existing few-shot and incremental learning methods across various scenarios.
Semantic segmentation models have two fundamental weaknesses: i) they require large training sets with costly pixel-level annotations, and ii) they have a static output space, constrained to the classes of the training set. Toward addressing both problems, we introduce a new task, Incremental Few-Shot Segmentation (iFSS). The goal of iFSS is to extend a pretrained segmentation model with new classes from few annotated images and without access to old training data. To overcome the limitations of existing models iniFSS, we propose Prototype-based Incremental Few-Shot Segmentation (PIFS) that couples prototype learning and knowledge distillation. PIFS exploits prototypes to initialize the classifiers of new classes, fine-tuning the network to refine its features representation. We design a prototype-based distillation loss on the scores of both old and new class prototypes to avoid overfitting and forgetting, and batch-renormalization to cope with non-i.i.d.few-shot data. We create an extensive benchmark for iFSS showing that PIFS outperforms several few-shot and incremental learning methods in all scenarios.