fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation
For researchers and practitioners in human-robot interaction, fARfetch addresses the challenge of maintaining AR content legibility and interaction effectiveness in large-scale outdoor environments, a known bottleneck in AR-HRC.
fARfetch enables collocated AR-HRC in large, visually diverse environments by integrating semantic mapping, context-aware world-in-miniature, and VLM-driven AR content adaptation. In a 30.5m outdoor inspection task, it achieved 66% faster completion times and significantly reduced workload (mental demand -43%, temporal demand -34%, frustration -66%) compared to a non-AR baseline.
Augmented Reality (AR) can improve collocated human-robot collaboration by making robot state and intent visible and enabling intuitive control, yet large, visually diverse environments like the outdoors challenge both interaction and content legibility, especially at long distances and beyond visual line of sight. We present fARfetch, an AR-HRC system that integrates (i) shared semantic environment mapping across an AR headset and robot that visualizes detected landmarks in AR to support landmark-grounded go-to commands, (ii) a context-aware world-in-miniature representation of the shared environment for fine-grained path authoring, and (iii) vision-language-model driven AR view management that jointly adapts virtual content color, size, and orientation to maintain legibility in large visually diverse environments. We implement fARfetch with a Meta Quest 3 headset and Unitree Go2 quadruped robot, and conduct a within-subjects user study (N=13) on a real-world large-scale (30.5m) outdoor inspection task. fARfetch yielded significantly faster completion times than a non-AR baseline (66%) and significantly lower workload in mental demand (-43%), temporal demand (-34%), and frustration (-66%). A custom legibility survey indicated fARfetch effectively maintained virtual content legibility in the large outdoor environment.