Open-Vocabulary Panoptic Segmentation Using BERT Pre-Training of Vision-Language Multiway Transformer Model
This work addresses the problem of generalizing to unlimited classes with limited data for segmentation, but it is incremental as it builds on existing vision-language models like BEiT-3.
The paper tackles open-vocabulary panoptic segmentation by proposing OMTSeg, which uses the BEiT-3 vision-language pre-trained model and cross-modal attention to improve performance, achieving favorable results against state-of-the-art models.
Open-vocabulary panoptic segmentation remains a challenging problem. One of the biggest difficulties lies in training models to generalize to an unlimited number of classes using limited categorized training data. Recent popular methods involve large-scale vision-language pre-trained foundation models, such as CLIP. In this paper, we propose OMTSeg for open-vocabulary segmentation using another large-scale vision-language pre-trained model called BEiT-3 and leveraging the cross-modal attention between visual and linguistic features in BEiT-3 to achieve better performance. Experiments result demonstrates that OMTSeg performs favorably against state-of-the-art models.