Continual Semantic Segmentation with Automatic Memory Sample Selection
This work addresses the challenge of efficiently managing memory in continual learning for semantic segmentation, which is incremental but improves performance for computer vision applications.
The paper tackles the problem of catastrophic forgetting in continual semantic segmentation by proposing an automatic memory sample selection mechanism that considers diversity and class performance, achieving state-of-the-art performance with a 12.54% improvement over the second-best method on Pascal-VOC 2012.
Continual Semantic Segmentation (CSS) extends static semantic segmentation by incrementally introducing new classes for training. To alleviate the catastrophic forgetting issue in CSS, a memory buffer that stores a small number of samples from the previous classes is constructed for replay. However, existing methods select the memory samples either randomly or based on a single-factor-driven handcrafted strategy, which has no guarantee to be optimal. In this work, we propose a novel memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way by considering comprehensive factors including sample diversity and class performance. Our mechanism regards the selection operation as a decision-making process and learns an optimal selection policy that directly maximizes the validation performance on a reward set. To facilitate the selection decision, we design a novel state representation and a dual-stage action space. Our extensive experiments on Pascal-VOC 2012 and ADE 20K datasets demonstrate the effectiveness of our approach with state-of-the-art (SOTA) performance achieved, outperforming the second-place one by 12.54% for the 6stage setting on Pascal-VOC 2012.