OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation
This addresses the practical challenge of expanding recognition capabilities in open-vocabulary segmentation for computer vision applications, though it is incremental as it builds on existing OVS methods.
The paper tackles the problem of open-vocabulary segmentation in continual learning scenarios where new datasets are collected over time, proposing ConOVS, a method based on a Mixture-of-Experts framework that outperforms existing approaches across pre-training, incremental, and zero-shot test datasets.
Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.