GaussExplorer: 3D Gaussian Splatting for Embodied Exploration and Reasoning
This addresses the challenge of spatial reasoning for embodied AI systems, though it appears incremental by building on prior 3DGS and VLM techniques.
The paper tackles the problem of enabling embodied agents to interpret complex compositional language queries in 3D scenes by integrating Vision-Language Models with 3D Gaussian Splatting, resulting in outperforming existing methods on benchmarks.
We present GaussExplorer, a framework for embodied exploration and reasoning built on 3D Gaussian Splatting (3DGS). While prior approaches to language-embedded 3DGS have made meaningful progress in aligning simple text queries with Gaussian embeddings, they are generally optimized for relatively simple queries and struggle to interpret more complex, compositional language queries. Alternative studies based on object-centric RGB-D structured memories provide spatial grounding but are constrained by pre-fixed viewpoints. To address these issues, GaussExplorer introduces Vision-Language Models (VLMs) on top of 3DGS to enable question-driven exploration and reasoning within 3D scenes. We first identify pre-captured images that are most correlated with the query question, and subsequently adjust them into novel viewpoints to more accurately capture visual information for better reasoning by VLMs. Experiments show that ours outperforms existing methods on several benchmarks, demonstrating the effectiveness of integrating VLM-based reasoning with 3DGS for embodied tasks.