Integrating Prior Observations for Incremental 3D Scene Graph Prediction
This addresses the limitation of existing methods that rely on complete scene reconstructions, making it more applicable for robotics and embodied AI in real-world environments, though it is incremental in nature.
The paper tackles the problem of predicting 3D semantic scene graphs in incremental settings by integrating prior observations and multi-modal information into a heterogeneous graph model, achieving scalable and generalizable results on the 3DSSG dataset.
3D semantic scene graphs (3DSSG) provide compact structured representations of environments by explicitly modeling objects, attributes, and relationships. While 3DSSGs have shown promise in robotics and embodied AI, many existing methods rely mainly on sensor data, not integrating further information from semantically rich environments. Additionally, most methods assume access to complete scene reconstructions, limiting their applicability in real-world, incremental settings. This paper introduces a novel heterogeneous graph model for incremental 3DSSG prediction that integrates additional, multi-modal information, such as prior observations, directly into the message-passing process. Utilizing multiple layers, the model flexibly incorporates global and local scene representations without requiring specialized modules or full scene reconstructions. We evaluate our approach on the 3DSSG dataset, showing that GNNs enriched with multi-modal information such as semantic embeddings (e.g., CLIP) and prior observations offer a scalable and generalizable solution for complex, real-world environments. The full source code of the presented architecture will be made available at https://github.com/m4renz/incremental-scene-graph-prediction.