What Holds Back Open-Vocabulary Segmentation?
This addresses the limitation of segmentation models for recognizing novel concepts, which is incremental as it builds on existing open-vocabulary approaches.
The paper tackles the problem of open-vocabulary segmentation, where models fail to recognize concepts outside their training taxonomy, and finds that performance has plateaued for nearly two years due to bottlenecks. It proposes oracle components using groundtruth information to identify and decouple these bottlenecks, providing empirical insights and suggesting approaches to unlock future research.
Standard segmentation setups are unable to deliver models that can recognize concepts outside the training taxonomy. Open-vocabulary approaches promise to close this gap through language-image pretraining on billions of image-caption pairs. Unfortunately, we observe that the promise is not delivered due to several bottlenecks that have caused the performance to plateau for almost two years. This paper proposes novel oracle components that identify and decouple these bottlenecks by taking advantage of the groundtruth information. The presented validation experiments deliver important empirical findings that provide a deeper insight into the failures of open-vocabulary models and suggest prominent approaches to unlock the future research.