Exploration of visual prompt in Grounded pre-trained open-set detection
This work addresses the problem of manual feedback requirements in text prompts for object detection, offering a more automated approach for researchers and practitioners in computer vision.
The paper tackles the limitation of text prompts in open-set object detection by proposing a visual prompt method that learns from few labeled images to generalize to new categories, achieving superior performance on the ODinW dataset compared to existing prompt learning methods.
Text prompts are crucial for generalizing pre-trained open-set object detection models to new categories. However, current methods for text prompts are limited as they require manual feedback when generalizing to new categories, which restricts their ability to model complex scenes, often leading to incorrect detection results. To address this limitation, we propose a novel visual prompt method that learns new category knowledge from a few labeled images, which generalizes the pre-trained detection model to the new category. To allow visual prompts to represent new categories adequately, we propose a statistical-based prompt construction module that is not limited by predefined vocabulary lengths, thus allowing more vectors to be used when representing categories. We further utilize the category dictionaries in the pre-training dataset to design task-specific similarity dictionaries, which make visual prompts more discriminative. We evaluate the method on the ODinW dataset and show that it outperforms existing prompt learning methods and performs more consistently in combinatorial inference.