MVDiff: Scalable and Flexible Multi-View Diffusion for 3D Object Reconstruction from Single-View
This work addresses a problem for 3D computer vision researchers and practitioners by improving multi-view generation for reconstruction, though it appears incremental as it builds on existing diffusion and transformer methods.
The paper tackles the challenge of generating consistent multiple views for 3D reconstruction from single images, proposing a framework that uses epipolar geometry constraints and multi-view attention to achieve 3D consistency, resulting in generated 3D meshes that surpass baseline methods in metrics like PSNR, SSIM, and LPIPS.
Generating consistent multiple views for 3D reconstruction tasks is still a challenge to existing image-to-3D diffusion models. Generally, incorporating 3D representations into diffusion model decrease the model's speed as well as generalizability and quality. This paper proposes a general framework to generate consistent multi-view images from single image or leveraging scene representation transformer and view-conditioned diffusion model. In the model, we introduce epipolar geometry constraints and multi-view attention to enforce 3D consistency. From as few as one image input, our model is able to generate 3D meshes surpassing baselines methods in evaluation metrics, including PSNR, SSIM and LPIPS.