Learning from Synthetic Data for Visual Grounding
This work addresses data scarcity for visual grounding tasks, offering a scalable synthetic data generation method, though it is incremental as it builds on existing models and benchmarks.
This paper tackles the problem of improving visual grounding in vision-and-language models by using synthetic training data, and finds that their proposed SynGround pipeline increases pointing game accuracy by up to 17.11% on benchmarks.
This paper extensively investigates the effectiveness of synthetic training data to improve the capabilities of vision-and-language models for grounding textual descriptions to image regions. We explore various strategies to best generate image-text pairs and image-text-box triplets using a series of pretrained models under different settings and varying degrees of reliance on real data. Through comparative analyses with synthetic, real, and web-crawled data, we identify factors that contribute to performance differences, and propose SynGround, an effective pipeline for generating useful synthetic data for visual grounding. Our findings show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models and offers the potential for arbitrarily large scale data generation. Particularly, data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively, across the RefCOCO+ and the Flickr30k benchmarks.