RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation
This work addresses efficiency and accuracy bottlenecks in 3D scene understanding for robotics applications like manipulation and navigation, representing an incremental improvement over existing methods.
The paper tackles the problem of low accuracy and speed in open-vocabulary 3D scene graph generation by proposing RAG-3DSG, which uses re-shot guided uncertainty estimation and retrieval-augmented generation to improve object recognition, resulting in significantly higher node captioning accuracy and a two-thirds reduction in mapping time compared to baseline methods.
Open-vocabulary 3D Scene Graph (3DSG) generation can enhance various downstream tasks in robotics, such as manipulation and navigation, by leveraging structured semantic representations. A 3DSG is constructed from multiple images of a scene, where objects are represented as nodes and relationships as edges. However, existing works for open-vocabulary 3DSG generation suffer from both low object-level recognition accuracy and speed, mainly due to constrained viewpoints, occlusions, and redundant surface density. To address these challenges, we propose RAG-3DSG to mitigate aggregation noise through re-shot guided uncertainty estimation and support object-level Retrieval-Augmented Generation (RAG) via reliable low-uncertainty objects. Furthermore, we propose a dynamic downsample-mapping strategy to accelerate cross-image object aggregation with adaptive granularity. Experiments on Replica dataset demonstrate that RAG-3DSG significantly improves node captioning accuracy in 3DSG generation while reducing the mapping time by two-thirds compared to the vanilla version.