Breathing New Life into 3D Assets with Generative Repainting
This work addresses the problem of enhancing 3D assets for artists and content creators by enabling efficient, non-learned integration of 2D generative priors, though it is incremental as it builds on existing diffusion and neural field techniques.
The paper tackles the challenge of applying 2D diffusion models to 3D assets by proposing a modular pipeline that combines pretrained 2D diffusion models with 3D neural radiance fields without additional learning, enabling generative repainting of legacy geometries like meshes. It demonstrates advantages through a large-scale study on the ShapeNetSem dataset, showing qualitative and quantitative improvements.
Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Broad adoption of these models is due to significant improvement in the quality of generations and efficient conditioning on various modalities, not just text. However, lifting the rich generative priors of these 2D models into 3D is challenging. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools and demonstrate their ability to work together in a non-learned fashion. Such modularity has the intrinsic advantage of eased partial upgrades, which became an important property in such a fast-paced domain. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools, and outputs a painted input geometry in several formats. We conduct a large-scale study on a wide range of objects and categories from the ShapeNetSem dataset and demonstrate the advantages of our approach, both qualitatively and quantitatively. Project page: https://www.obukhov.ai/repainting_3d_assets