Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversion
This addresses a novel task in 3D generation for applications like virtual reality or content creation, though it appears incremental as it builds on existing diffusion models.
The paper tackles the problem of converting a 3D scene to another based on text instructions, achieving higher quality conversions than baseline methods through quantitative and qualitative evaluations.
We propose a high-quality 3D-to-3D conversion method, Instruct 3D-to-3D. Our method is designed for a novel task, which is to convert a given 3D scene to another scene according to text instructions. Instruct 3D-to-3D applies pretrained Image-to-Image diffusion models for 3D-to-3D conversion. This enables the likelihood maximization of each viewpoint image and high-quality 3D generation. In addition, our proposed method explicitly inputs the source 3D scene as a condition, which enhances 3D consistency and controllability of how much of the source 3D scene structure is reflected. We also propose dynamic scaling, which allows the intensity of the geometry transformation to be adjusted. We performed quantitative and qualitative evaluations and showed that our proposed method achieves higher quality 3D-to-3D conversions than baseline methods.