Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs
For robotics and embodied AI, this enables real-time queryable 3D scene understanding without waiting for full reconstruction.
The paper proposes an asynchronous architecture for incremental 3D scene graph construction that allows querying during exploration, achieving state-of-the-art results on visual grounding benchmarks (15.3-18.8 A@0.25 improvement) while matching or outperforming existing methods on semantic segmentation.
Open-vocabulary 3D scene graph methods typically operate in two stages: first reconstruct, then enrich with vision-language models, leaving the graph unqueryable during exploration. We argue that this sequential coupling is unnecessary and propose an asynchronous architecture in which lightweight online mapping runs concurrently with heavyweight semantic refinement. A probabilistic voxel-based backbone maintains stable object identities incrementally, while background VLM agents progressively enrich the graph. This framework resolves duplicate object tracks through semantic loop closure, attaches fine-grained visual attributes and derives spatial relations between objects. A multi-target frame scheduler amortizes VLM cost by selecting a small set of informative frames that jointly cover multiple targets. The resulting scene graph is queryable during exploration and grows in semantic richness over time. Our method matches or outperforms existing open-vocabulary 3D scene graph methods on semantic segmentation (ScanNet, Replica) and surpasses the prior state-of-the-art across three visual grounding benchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25. Project page: https://denizbickici.github.io/thinkgraphs/