Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation
This addresses the challenge of high-quality 3D generation for applications in computer graphics and AI, representing a novel method rather than an incremental improvement.
The paper tackles the problem of direct 3D generation constrained by scarce and low-fidelity 3D datasets by introducing Zero-1-to-G, which enables direct single-view generation on Gaussian splats using pretrained 2D diffusion models, achieving superior performance in 3D object generation on synthetic and in-the-wild datasets.
Recent advances in 2D image generation have achieved remarkable quality,largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct image-to-3D generative model to effectively utilize pretrained 2D diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.