Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions
This work addresses scene understanding for computer vision applications, offering a domain-agnostic retrieval method, but it is incremental as it builds on existing techniques like PointNet and GCNs.
The paper tackles 3D scene understanding by proposing a learned method to regress scene graphs from point clouds, using a novel architecture based on PointNet and Graph Convolutional Networks, and introduces the 3DSSG dataset for evaluation.
Scene understanding has been of high interest in computer vision. It encompasses not only identifying objects in a scene, but also their relationships within the given context. With this goal, a recent line of works tackles 3D semantic segmentation and scene layout prediction. In our work we focus on scene graphs, a data structure that organizes the entities of a scene in a graph, where objects are nodes and their relationships modeled as edges. We leverage inference on scene graphs as a way to carry out 3D scene understanding, mapping objects and their relationships. In particular, we propose a learned method that regresses a scene graph from the point cloud of a scene. Our novel architecture is based on PointNet and Graph Convolutional Networks (GCN). In addition, we introduce 3DSSG, a semi-automatically generated dataset, that contains semantically rich scene graphs of 3D scenes. We show the application of our method in a domain-agnostic retrieval task, where graphs serve as an intermediate representation for 3D-3D and 2D-3D matching.