Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality
This addresses the problem of inefficient and complex agent interactions in VR for users, but it appears incremental as it builds on existing LLM agent methods with optimizations for a specific domain.
The paper tackles the challenge of deploying large language models as agents in virtual reality by proposing Voice2Action, a framework that hierarchically processes voice and text commands for real-time interaction, showing improved efficiency and accuracy in experiments with synthetic data in an urban engineering VR environment.
Large Language Models (LLMs) are trained and aligned to follow natural language instructions with only a handful of examples, and they are prompted as task-driven autonomous agents to adapt to various sources of execution environments. However, deploying agent LLMs in virtual reality (VR) has been challenging due to the lack of efficiency in online interactions and the complex manipulation categories in 3D environments. In this work, we propose Voice2Action, a framework that hierarchically analyzes customized voice signals and textual commands through action and entity extraction and divides the execution tasks into canonical interaction subsets in real-time with error prevention from environment feedback. Experiment results in an urban engineering VR environment with synthetic instruction data show that Voice2Action can perform more efficiently and accurately than approaches without optimizations.