Multimodal Language Models Cannot Spot Spatial Inconsistencies
This addresses a key requirement for models to understand physical reality, revealing a critical limitation in current MLLMs for tasks involving spatial reasoning.
The paper tackled the problem of multimodal large language models (MLLMs) struggling with spatial consistency in 3D geometry across multiple views, showing that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability in identifying objects that violate 3D motion consistency.
Spatial consistency is a fundamental property of the visual world and a key requirement for models that aim to understand physical reality. Despite recent advances, multimodal large language models (MLLMs) often struggle to reason about 3D geometry across multiple views. Rather than asking models to describe scene attributes, we introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency. We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes, enabling systematic evaluation of this capability. Our results show that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability across different scene attributes, revealing a fragile and incomplete understanding of 3D structure. We hope our findings underscore the need for approaches that develop a more deeply grounded understanding of the physical world.