Enhancing Monocular 3D Scene Completion with Diffusion Model
This work addresses the challenge of monocular 3D scene completion for applications like virtual reality and robotics, offering an incremental improvement by reducing reliance on multi-view inputs.
The paper tackles the problem of 3D scene reconstruction from a single image, which is limited by traditional multi-view methods, and introduces FlashDreamer, a method that uses a vision-language model and diffusion model to generate multi-view images for reconstruction, achieving effective and robust results without further training.
3D scene reconstruction is essential for applications in virtual reality, robotics, and autonomous driving, enabling machines to understand and interact with complex environments. Traditional 3D Gaussian Splatting techniques rely on images captured from multiple viewpoints to achieve optimal performance, but this dependence limits their use in scenarios where only a single image is available. In this work, we introduce FlashDreamer, a novel approach for reconstructing a complete 3D scene from a single image, significantly reducing the need for multi-view inputs. Our approach leverages a pre-trained vision-language model to generate descriptive prompts for the scene, guiding a diffusion model to produce images from various perspectives, which are then fused to form a cohesive 3D reconstruction. Extensive experiments show that our method effectively and robustly expands single-image inputs into a comprehensive 3D scene, extending monocular 3D reconstruction capabilities without further training. Our code is available https://github.com/CharlieSong1999/FlashDreamer/tree/main.