Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations
This addresses interaction challenges for MR users by providing a new paradigm that simplifies object selection and manipulation in complex environments.
The paper tackles the difficulty of interacting with real-world objects in Mixed Reality (MR) when they are crowded, distant, or occluded by introducing Reality Proxy, a system that uses abstract representations (proxies) to decouple interaction from physical constraints, enabling novel interactions like skimming and filtering without new gestures, with an expert evaluation suggesting its utility and usability.
Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.