Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
This work addresses the challenge of 3D reasoning for AI systems in robotics and augmented reality, representing a novel method rather than an incremental improvement.
The paper tackles the problem of spatial understanding and viewpoint-aware reasoning in multimodal large language models by introducing Loc3R-VLM, a framework that enhances 2D vision-language models with 3D capabilities from monocular video, achieving state-of-the-art performance in language-based localization and 3D question-answering benchmarks.
Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm