Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
This work addresses the challenge of 3D scene understanding for AI systems by reducing reliance on costly 3D data, though it builds incrementally on prior video-based approaches.
The paper tackles the problem of enabling Multimodal Large Language Models (MLLMs) to understand 3D scenes directly from video data without requiring additional 3D inputs like point clouds, achieving competitive results with a 4B model that surpasses Gemini-1.5-Pro in VSI-Bench evaluations.
Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.