LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
This addresses the challenge of designing interactive Mixed Reality worlds with scarce training data, though it appears incremental as it builds on existing LLM and game engine technologies.
The paper tackles the problem of real-time creation and modification of interactive Mixed Reality experiences using LLMs, achieving a 4x reduction in average error rate compared to GPT-4 and positive usability feedback from participants.
We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR's cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.