Multi-Turn Multi-Agent Dialogue for Collaborative Reconstruction Improves VLM Performance on Spatial Reasoning, But Only Barely
This work identifies a critical limitation in visual spatial grounding and grounded instruction generation for collaborative VLM agents, impacting the development of more capable robots for human-collaborative tasks.
This paper investigates the ability of Vision-Language Models (VLMs) to perform collaborative structure-building tasks that require spatial reasoning. It finds that while detailed text representations and decomposed image representations can slightly improve reconstruction success, VLMs still struggle significantly with visual spatial grounding and generating grounded instructions for collaborative agents.
Robots operating in diverse environments rely on visual input to interpret objects and spatial layouts. In human-collaborative tasks, they are expected to communicate this understanding through language. Vision-language models (VLMs) support robotic tasks involving visual interpretation, question answering, and instruction following, but their capabilities in collaborative dialogue tasks requiring spatial reasoning remain underexplored. We study this gap through a collaborative structure-building task that combines visual interpretation, grounding, language-guided interaction, and action generation. We develop a framework in which VLMs use dialogue to reconstruct a target structure from visual and textual inputs. We evaluate open-weight and closed VLMs across interaction settings, input modalities, and image representations. Results show that spatial reasoning over visual representations remains difficult for the evaluated VLMs. Detailed text representations of the target yield higher reconstruction success across modality conditions, while decomposed image representations improve performance. These findings reveal limits in visual spatial grounding and grounded instruction generation for collaborative VLM agents.