Open-Set Recognition in the Age of Vision-Language Models
This addresses a critical vulnerability in VLMs for open-set recognition, which is important for applications requiring robust perception in unpredictable environments, though it is incremental in refining existing definitions and methods.
The paper tackles the problem that vision-language models (VLMs) are not inherently open-set models, as they misclassify objects not in their query set, leading to low precision or recall, and shows that increasing query set size worsens performance. It establishes a new benchmark and evaluates baseline approaches to address this issue.
Are vision-language models (VLMs) for open-vocabulary perception inherently open-set models because they are trained on internet-scale datasets? We answer this question with a clear no - VLMs introduce closed-set assumptions via their finite query set, making them vulnerable to open-set conditions. We systematically evaluate VLMs for open-set recognition and find they frequently misclassify objects not contained in their query set, leading to alarmingly low precision when tuned for high recall and vice versa. We show that naively increasing the size of the query set to contain more and more classes does not mitigate this problem, but instead causes diminishing task performance and open-set performance. We establish a revised definition of the open-set problem for the age of VLMs, define a new benchmark and evaluation protocol to facilitate standardised evaluation and research in this important area, and evaluate promising baseline approaches based on predictive uncertainty and dedicated negative embeddings on a range of open-vocabulary VLM classifiers and object detectors.