Perception-Aware Multimodal Spatial Reasoning from Monocular Images
This work provides a significant improvement in spatial reasoning from monocular images for autonomous driving, particularly for scenarios with challenging perception conditions, by enhancing VLM capabilities.
This paper addresses the challenge of fine-grained geometric perception in Vision-Language Models (VLMs) for spatial reasoning from monocular images, especially under scale variation and ambiguous object appearance. The proposed framework, which represents referred objects using Visual Reference Tokens (VRTs) and processes visual and textual reasoning jointly, achieves substantial improvements on the SURDS benchmark, outperforming previous methods by a large margin across single-object and multi-object tasks.
Spatial reasoning from monocular images is essential for autonomous driving, yet current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception, particularly under large scale variation and ambiguous object appearance. We propose a simple yet effective perception-aware multimodal reasoning framework that equips VLMs with explicit object-centric grounding ability. Instead of relying on textual bounding-box outputs, each referred object is represented using all Visual Reference Tokens (VRTs) within its spatial extent, enabling visual evidence and textual reasoning to be processed jointly in a unified token space. To further strengthen cross-modal interaction, we construct a Multimodal Chain-of-Thought (MM-CoT) dataset that injects aligned visual and textual reasoning signals. A deterministic ordering strategy is introduced to make supervision over inherently unordered VRT sets fully compatible with the VLM's autoregressive next-token prediction. With only standard supervised fine-tuning, our method achieves substantial improvements on the SURDS benchmark, outperforming previous approaches - including those using RL-based post-training - by a large margin across both single-object and multi-object tasks. These results demonstrate that accurate perception and multimodal reasoning are mutually reinforcing, and together form the key to robust spatial understanding in challenging monocular driving scenarios.