Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation
This work addresses inefficiencies in evaluating and pre-training open-vocabulary SGG models, which is important for researchers in computer vision and AI, though it appears incremental as it builds on existing VLM advancements.
The paper tackles the problem of evaluating open-vocabulary scene graph generation (SGG) by proposing a new reference-free metric to assess vision-language models (VLMs) for relation prediction, and introduces a method for generating high-quality synthetic data through region-specific prompt tuning to improve model generalization.
Scene Graph Generation (SGG) encodes visual relationships between objects in images as graph structures. Thanks to the advances of Vision-Language Models (VLMs), the task of Open-Vocabulary SGG has been recently proposed where models are evaluated on their functionality to learn a wide and diverse range of relations. Current benchmarks in SGG, however, possess a very limited vocabulary, making the evaluation of open-source models inefficient. In this paper, we propose a new reference-free metric to fairly evaluate the open-vocabulary capabilities of VLMs for relation prediction. Another limitation of Open-Vocabulary SGG is the reliance on weakly supervised data of poor quality for pre-training. We also propose a new solution for quickly generating high-quality synthetic data through region-specific prompt tuning of VLMs. Experimental results show that pre-training with this new data split can benefit the generalization capabilities of Open-Voc SGG models.