The Spatial Blindspot of Vision-Language Models
This addresses a bottleneck for applications like robotics and embodied AI, but is incremental as it builds on existing VLM architectures.
The paper tackles the problem of vision-language models lacking spatial awareness due to flattened image encoders, and finds that using alternative objectives and 2D positional encodings improves spatial reasoning on benchmarks.
Vision-language models (VLMs) have advanced rapidly, but their ability to capture spatial relationships remains a blindspot. Current VLMs are typically built with contrastive language-image pretraining (CLIP) style image encoders. The training recipe often flattens images into 1D patch sequences, discarding the 2D structure necessary for spatial reasoning. We argue that this lack of spatial awareness is a missing dimension in VLM design and a bottleneck for applications requiring spatial grounding, such as robotics and embodied AI. To address this, we investigate (i) image encoders trained with alternative objectives and (ii) 2D positional encodings. Our experiments show that these architectural choices can lead to improved spatial reasoning on several benchmarks.