Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation
This work solves the ill-posed task of single-image-to-3D mesh generation for applications in computer graphics and vision, offering plug-and-play modules that enhance existing methods.
The paper tackles the problem of generating high-quality 3D meshes from a single image by addressing inconsistencies in multiview images and low fidelity in existing methods, achieving state-of-the-art performance with significant improvements as verified by experiments.
Generating 3D meshes from a single image is an important but ill-posed task. Existing methods mainly adopt 2D multiview diffusion models to generate intermediate multiview images, and use the Large Reconstruction Model (LRM) to create the final meshes. However, the multiview images exhibit local inconsistencies, and the meshes often lack fidelity to the input image or look blurry. We propose Fancy123, featuring two enhancement modules and an unprojection operation to address the above three issues, respectively. The appearance enhancement module deforms the 2D multiview images to realign misaligned pixels for better multiview consistency. The fidelity enhancement module deforms the 3D mesh to match the input image. The unprojection of the input image and deformed multiview images onto LRM's generated mesh ensures high clarity, discarding LRM's predicted blurry-looking mesh colors. Extensive qualitative and quantitative experiments verify Fancy123's SoTA performance with significant improvement. Also, the two enhancement modules are plug-and-play and work at inference time, allowing seamless integration into various existing single-image-to-3D methods. Code at: https://github.com/YuQiao0303/Fancy123