World Mouse: Exploring Interactions with a Cross-Reality Cursor
This addresses the problem of limited precision and reach in XR interactions for users, though it appears incremental as it builds on existing virtual cursor concepts by incorporating physical objects.
The paper tackled the challenge of interacting with blended physical and virtual representations in Extended Reality (XR) by introducing World Mouse, a cross-reality cursor that reinterprets the 2D desktop mouse for 3D scenes, enabling seamless interactions across real and virtual environments.
As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.