Diffusion Time-step Curriculum for One Image to 3D Generation
This work addresses a key challenge in 3D generation from single images for applications in graphics and AI, offering an incremental improvement over existing SDS-based methods.
The paper tackles the problem of generating 3D objects from a single image using score distillation sampling (SDS), which often suffers from geometric artifacts and texture saturation, by proposing a diffusion time-step curriculum that treats knowledge distillation in a coarse-to-fine manner, resulting in multi-view consistent, high-quality, and diverse 3D assets as demonstrated on benchmarks like NeRF4 and RealFusion15.
Score distillation sampling~(SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a \textbf{single} image. It leverages pre-trained 2D diffusion models as teacher to guide the reconstruction of student 3D models. Despite their remarkable success, SDS-based methods often encounter geometric artifacts and texture saturation. We find out the crux is the overlooked indiscriminate treatment of diffusion time-steps during optimization: it unreasonably treats the student-teacher knowledge distillation to be equal at all time-steps and thus entangles coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the teacher and student models collaborating with the time-step curriculum in a coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and Level50 benchmark demonstrate that DTC123 can produce multi-view consistent, high-quality, and diverse 3D assets. Codes and more generation demos will be released in https://github.com/yxymessi/DTC123.