Continual Learning for Image Segmentation with Dynamic Query
This addresses the challenge of maintaining performance when incrementally adding new classes in image segmentation, which is crucial for applications like autonomous driving and medical imaging, though it appears incremental as it builds on existing continual learning techniques.
The paper tackles the problem of catastrophic forgetting and background shift in continual learning for image segmentation by proposing CISDQ, a method that uses dynamic queries and knowledge distillation, achieving state-of-the-art performance with 4.4% and 2.9% mIoU improvements on specific ADE dataset settings.
Image segmentation based on continual learning exhibits a critical drop of performance, mainly due to catastrophic forgetting and background shift, as they are required to incorporate new classes continually. In this paper, we propose a simple, yet effective Continual Image Segmentation method with incremental Dynamic Query (CISDQ), which decouples the representation learning of both old and new knowledge with lightweight query embedding. CISDQ mainly includes three contributions: 1) We define dynamic queries with adaptive background class to exploit past knowledge and learn future classes naturally. 2) CISDQ proposes a class/instance-aware Query Guided Knowledge Distillation strategy to overcome catastrophic forgetting by capturing the inter-class diversity and intra-class identity. 3) Apart from semantic segmentation, CISDQ introduce the continual learning for instance segmentation in which instance-wise labeling and supervision are considered. Extensive experiments on three datasets for two tasks (i.e., continual semantic and instance segmentation are conducted to demonstrate that CISDQ achieves the state-of-the-art performance, specifically, obtaining 4.4% and 2.9% mIoU improvements for the ADE 100-10 (6 steps) setting and ADE 100-5 (11 steps) setting.