Do 3D Large Language Models Really Understand 3D Spatial Relationships?
This work addresses the need for robust benchmarks and training strategies to advance genuine 3D vision-language understanding, particularly for researchers and developers in AI and computer vision.
The paper tackles the problem of evaluating 3D Large Language Models (3D-LLMs) for spatial understanding by showing that fine-tuning on text-only data matches or surpasses them on the SQA3D benchmark, indicating it may not detect textual shortcuts. They introduce Real-3DQA, a more rigorous benchmark, and a 3D-reweighted training objective that substantially enhances 3D-LLMs' performance in spatial reasoning tasks.
Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.