Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
This work addresses a critical limitation in vision-language models for applications requiring spatial understanding, but it is incremental as it diagnoses rather than solves the issue.
The study tackled the problem of vision-language models struggling with basic spatial reasoning, such as relative position and layout, by conducting a controlled diagnostic analysis within the LLaVA framework, finding consistent performance gaps across models that indicate encoder objectives and positional structure influence spatial behavior but do not fully resolve it.
Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.