PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion
This addresses the challenge of 3D content creation for applications like gaming and VR, offering a fast and efficient solution, though it builds incrementally on existing diffusion methods.
The paper tackles the problem of generating 3D shapes from text prompts by leveraging pre-trained 2D diffusion models, achieving high-quality results in only 3 minutes per shape and outperforming existing 3D generative models significantly.
Diffusion models trained on large-scale text-image datasets have demonstrated a strong capability of controllable high-quality image generation from arbitrary text prompts. However, the generation quality and generalization ability of 3D diffusion models is hindered by the scarcity of high-quality and large-scale 3D datasets. In this paper, we present PI3D, a framework that fully leverages the pre-trained text-to-image diffusion models' ability to generate high-quality 3D shapes from text prompts in minutes. The core idea is to connect the 2D and 3D domains by representing a 3D shape as a set of Pseudo RGB Images. We fine-tune an existing text-to-image diffusion model to produce such pseudo-images using a small number of text-3D pairs. Surprisingly, we find that it can already generate meaningful and consistent 3D shapes given complex text descriptions. We further take the generated shapes as the starting point for a lightweight iterative refinement using score distillation sampling to achieve high-quality generation under a low budget. PI3D generates a single 3D shape from text in only 3 minutes and the quality is validated to outperform existing 3D generative models by a large margin.