FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesis
This addresses the challenge of creating immersive 3D content from limited 2D inputs for applications in virtual reality or gaming, though it appears incremental as it builds on existing video models and depth estimation techniques.
The paper tackles the problem of generating flexible-view 3D scenes, such as 360° rotations and zooming, from single images by introducing FlexWorld, a framework that uses a video-to-video diffusion model and progressive expansion to achieve superior visual quality compared to state-of-the-art methods.
Generating flexible-view 3D scenes, including 360° rotation and zooming, from single images is challenging due to a lack of 3D data. To this end, we introduce FlexWorld, a novel framework consisting of two key components: (1) a strong video-to-video (V2V) diffusion model to generate high-quality novel view images from incomplete input rendered from a coarse scene, and (2) a progressive expansion process to construct a complete 3D scene. In particular, leveraging an advanced pre-trained video model and accurate depth-estimated training pairs, our V2V model can generate novel views under large camera pose variations. Building upon it, FlexWorld progressively generates new 3D content and integrates it into the global scene through geometry-aware scene fusion. Extensive experiments demonstrate the effectiveness of FlexWorld in generating high-quality novel view videos and flexible-view 3D scenes from single images, achieving superior visual quality under multiple popular metrics and datasets compared to existing state-of-the-art methods. Qualitatively, we highlight that FlexWorld can generate high-fidelity scenes with flexible views like 360° rotations and zooming. Project page: https://ml-gsai.github.io/FlexWorld.