SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
This work addresses a key limitation in vision-language models for applications like robotics and navigation, though it is incremental as it focuses on data augmentation rather than a new model paradigm.
The paper tackled the problem of vision-language models struggling with spatial reasoning by constructing a synthetic VQA dataset of 455k samples, resulting in up to a 49% performance gain on spatial reasoning benchmarks while maintaining general task performance.
Vision-language models (VLMs) work well in tasks ranging from image captioning to visual question answering (VQA), yet they struggle with spatial reasoning, a key skill for understanding our physical world that humans excel at. We find that spatial relations are generally rare in widely used VL datasets, with only a few being well represented, while most form a long tail of underrepresented relations. This gap leaves VLMs ill-equipped to handle diverse spatial relationships. To bridge it, we construct a synthetic VQA dataset focused on spatial reasoning generated from hyper-detailed image descriptions in Localized Narratives, DOCCI, and PixMo-Cap. Our dataset consists of 455k samples containing 3.4 million QA pairs. Trained on this dataset, our Spatial-Reasoning Enhanced (SpaRE) VLMs show strong improvements on spatial reasoning benchmarks, achieving up to a 49% performance gain on the What's Up benchmark, while maintaining strong results on general tasks. Our work narrows the gap between human and VLM spatial reasoning and makes VLMs more capable in real-world tasks such as robotics and navigation.