Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation
This work addresses the need for more natural and efficient user interactions in VR environments, offering a novel interface that could benefit VR developers and users, though it appears incremental as it builds on existing LLM and VR technologies.
The paper tackles the problem of intuitive object manipulation in virtual reality by proposing VR Mover, an LLM-based system that interprets vocal instructions and pointing gestures, resulting in enhanced usability, reduced workload and arm fatigue, and improved performance in multi-object manipulation tasks.
In our daily lives, we can naturally convey instructions for the spatial manipulation of objects using words and gestures. Transposing this form of interaction into virtual reality (VR) object manipulation can be beneficial. We propose VR Mover, an LLM-empowered solution that can understand and interpret the user's vocal instruction to support object manipulation. By simply pointing and speaking, the LLM can manipulate objects without structured input. Our user study demonstrates that VR Mover enhances user usability, overall experience and performance on multi-object manipulation, while also reducing workload and arm fatigue. Users prefer the proposed natural interface for broad movements and may complementarily switch to gizmos or virtual hands for finer adjustments. These findings are believed to contribute to design implications for future LLM-based object manipulation interfaces, highlighting the potential for more intuitive and efficient user interactions in VR environments.