Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph
This work addresses the challenge of achieving natural AR interactions in diverse indoor environments for users, though it appears incremental as it builds on existing scene graph and semantic mapping techniques.
The paper tackles the problem of making augmented reality (AR) context-aware in indoor scenes by using 3D scene graphs to represent relationships among objects, enabling natural interaction between virtual and real worlds. The results show effectiveness in providing context-aware AR experiences across various real scenes, with evaluations including quantitative performance of oriented bounding box estimation and subjective assessments.
In this paper, we present Retargetable AR, a novel AR framework that yields an AR experience that is aware of scene contexts set in various real environments, achieving natural interaction between the virtual and real worlds. To this end, we characterize scene contexts with relationships among objects in 3D space, not with coordinates transformations. A context assumed by an AR content and a context formed by a real environment where users experience AR are represented as abstract graph representations, i.e. scene graphs. From RGB-D streams, our framework generates a volumetric map in which geometric and semantic information of a scene are integrated. Moreover, using the semantic map, we abstract scene objects as oriented bounding boxes and estimate their orientations. With such a scene representation, our framework constructs, in an online fashion, a 3D scene graph characterizing the context of a real environment for AR. The correspondence between the constructed graph and an AR scene graph denoting the context of AR content provides a semantically registered content arrangement, which facilitates natural interaction between the virtual and real worlds. We performed extensive evaluations on our prototype system through quantitative evaluation of the performance of the oriented bounding box estimation, subjective evaluation of the AR content arrangement based on constructed 3D scene graphs, and an online AR demonstration. The results of these evaluations showed the effectiveness of our framework, demonstrating that it can provide a context-aware AR experience in a variety of real scenes.