Steps towards prompt-based creation of virtual worlds
This work addresses the challenge of AI-assisted co-creation in VR for developers and users, but it is incremental as it builds on existing code generation models.
The paper tackles the problem of creating virtual worlds by applying large language models trained for code generation, demonstrating that prompt-based methods can accelerate in-VR level editing and integrate into gameplay, as shown with Codex VR Pong which enables non-deterministic game mechanics and interactions between 3D objects.
Large language models trained for code generation can be applied to speaking virtual worlds into existence (creating virtual worlds). In this work we show that prompt-based methods can both accelerate in-VR level editing, as well as can become part of gameplay rather than just part of game development. As an example, we present Codex VR Pong which shows non-deterministic game mechanics using generative processes to not only create static content but also non-trivial interactions between 3D objects. This demonstration naturally leads to an integral discussion on how one would evaluate and benchmark experiences created by generative models - as there are no qualitative or quantitative metrics that apply in these scenarios. We conclude by discussing impending challenges of AI-assisted co-creation in VR.