SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
This addresses the problem of catastrophic forgetting in semantic segmentation for incremental learning scenarios, though it appears incremental as it builds on existing knowledge distillation techniques.
The paper tackles catastrophic forgetting in class-incremental semantic segmentation by proposing SSUL-M, which defines unknown classes within backgrounds, freezes networks, and uses exemplar memory, achieving significantly better performance than recent state-of-the-art baselines on standard benchmarks.
This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.