Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop
This addresses the challenge of generating high-quality, controllable videos for users in creative or simulation domains, though it appears incremental by integrating existing tools like LLMs and 3D generators.
The paper tackles the problem of artifacts like temporal inconsistency in video generation by proposing Scene Copilot, a framework that combines large language models with a procedural 3D scene generator to enable easy creation of diverse photorealistic scenes, with experiments demonstrating its capability in customizing 3D scenes and video generation.
Video generation has achieved impressive quality, but it still suffers from artifacts such as temporal inconsistency and violation of physical laws. Leveraging 3D scenes can fundamentally resolve these issues by providing precise control over scene entities. To facilitate the easy generation of diverse photorealistic scenes, we propose Scene Copilot, a framework combining large language models (LLMs) with a procedural 3D scene generator. Specifically, Scene Copilot consists of Scene Codex, BlenderGPT, and Human in the loop. Scene Codex is designed to translate textual user input into commands understandable by the 3D scene generator. BlenderGPT provides users with an intuitive and direct way to precisely control the generated 3D scene and the final output video. Furthermore, users can utilize Blender UI to receive instant visual feedback. Additionally, we have curated a procedural dataset of objects in code format to further enhance our system's capabilities. Each component works seamlessly together to support users in generating desired 3D scenes. Extensive experiments demonstrate the capability of our framework in customizing 3D scenes and video generation.