Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality
This addresses the challenge of selecting multiple occluded objects in virtual reality for users, though it appears incremental as it builds on existing multimodal interaction concepts with LLM assistance.
The researchers tackled the problem of selecting multiple occluded objects in virtual reality by developing AssistVR, a multimodal speech and raycast interaction technique powered by large language models. In a user study with 24 participants, AssistVR outperformed a baseline mini-map technique for multiple target objects, even when objects were difficult to reference verbally.
Selection of occluded objects is a challenging problem in virtual reality, even more so if multiple objects are involved. With the advent of new artificial intelligence technologies, we explore the possibility of leveraging large language models to assist multi-object selection tasks in virtual reality via a multimodal speech and raycast interaction technique. We validate the findings in a comparative user study (n=24), where participants selected target objects in a virtual reality scene with different levels of scene perplexity. The performance metrics and user experience metrics are compared against a mini-map based occluded object selection technique that serves as the baseline. Results indicate that the introduced technique, AssistVR, outperforms the baseline technique when there are multiple target objects. Contrary to the common belief for speech interfaces, AssistVR was able to outperform the baseline even when the target objects were difficult to reference verbally. This work demonstrates the viability and interaction potential of an intelligent multimodal interactive system powered by large laguage models. Based on the results, we discuss the implications for design of future intelligent multimodal interactive systems in immersive environments.