Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models
This work addresses the lack of comprehensive benchmarks for 3D spatial reasoning in AI models, which is crucial for applications like robotics and autonomous systems, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating 6D spatial reasoning in large multimodal models by introducing Spatial457, a synthetic dataset with cascading evaluation tasks, and found that model performance declines as task complexity increases, particularly in 3D and 6D tasks, with a Relative Performance Dropping Rate quantifying these weaknesses.
Although large multimodal models (LMMs) have demonstrated remarkable capabilities in visual scene interpretation and reasoning, their capacity for complex and precise 3-dimensional spatial reasoning remains uncertain. Existing benchmarks focus predominantly on 2D spatial understanding and lack a framework to comprehensively evaluate 6D spatial reasoning across varying complexities. To address this limitation, we present Spatial457, a scalable and unbiased synthetic dataset designed with 4 key capability for spatial reasoning: multi-object recognition, 2D location, 3D location, and 3D orientation. We develop a cascading evaluation structure, constructing 7 question types across 5 difficulty levels that range from basic single object recognition to our new proposed complex 6D spatial reasoning tasks. We evaluated various large multimodal models (LMMs) on PulseCheck457, observing a general decline in performance as task complexity increases, particularly in 3D reasoning and 6D spatial tasks. To quantify these challenges, we introduce the Relative Performance Dropping Rate (RPDR), highlighting key weaknesses in 3D reasoning capabilities. Leveraging the unbiased attribute design of our dataset, we also uncover prediction biases across different attributes, with similar patterns observed in real-world image settings. The code and data are released in https://github.com/XingruiWang/Spatial457.