Semantic Reality: Interactive Context-Aware Visualization of Inter-Object Relationships in Augmented Reality
This addresses a problem for AR users in planning, comparison, and assembly tasks by providing an incremental improvement over existing single-object systems.
The paper tackles the challenge of supporting tasks that depend on relationships among multiple objects in augmented reality by introducing Semantic Reality, an AR system that surfaces and visualizes inter-object connectivity interactively. In a study, participants reported clearer understanding and higher engagement compared to a single-object baseline, without increased workload.
Bridging the physical and digital world through interaction remains a core challenge in augmented reality (AR). Existing systems target single objects, limiting support for planning, comparison, and assembly tasks that depend on relationships among multiple items. We present Semantic Reality, an AR system focused on surfacing inter-object connectivity and making it interactive. Leveraging multimodal reasoning, spatial anchoring, and physical action recognition, Semantic Reality maintains a persistent model of objects around the user and their relationships. Connections are visualized in-situ to highlight compatibility, reveal next steps, and reduce ambiguity during tasks. We contribute a connectivity-centered interaction paradigm and a system architecture that couples anchor tracking, action sensing, and model inference to construct a live connectivity graph. In an exploratory study comparing Semantic Reality to a single-object baseline, participants reported clearer inter-object understanding and higher engagement and satisfaction, without increased workload. A scenario study illustrates where connectivity aids planning, sequencing, and disambiguation.