AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems
This work tackles the complex problem of automating 3D game generation in commercial engines for game developers, offering a novel approach to streamline the development process.
This paper introduces AutoUE, a multi-agent system that automates the end-to-end generation of 3D games within Unreal Engine, encompassing model retrieval, scene generation, gameplay code synthesis, and automated testing. It addresses challenges like tool-use hallucinations in LLMs by integrating retrieval-augmented generation with UE documentation and incorporating game design patterns and engine constraints into code generation.
Automatically generating 3D games in commercial game engines remains a non-trivial challenge, as it involves complex engine-related workflows for generating assets such as scenes, blueprints, and code. To address this challenge, we propose a novel multi-agent system, AutoUE, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. In order to mitigate tool-use hallucinations in LLMs, we introduce a retrieval-augmented generation mechanism that grounds agents with relevant UE tool documentation. Additionally, we incorporate game design patterns and engine constraints into the code generation process to ensure the generation of correct and robust code. Furthermore, we design an automated play-testing pipeline that generates and executes runtime test commands, enabling systematic evaluation of dynamic behaviors. Finally, we construct a game generation dataset and conduct a series of experiments that demonstrate AutoUE's ability to generate 3D games end-to-end, and validate the effectiveness of these designs.