Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors
This addresses the challenge of 3D-aware editing for open-domain images without needing synthetic multi-view datasets, though it builds incrementally on existing diffusion priors.
The paper tackles the problem of enabling 3D manipulations like rotation and translation on single images, achieving high-quality edits with large viewpoint transformations and consistency using pre-trained diffusion models.
We propose a novel image editing technique that enables 3D manipulations on single images, such as object rotation and translation. Existing 3D-aware image editing approaches typically rely on synthetic multi-view datasets for training specialized models, thus constraining their effectiveness on open-domain images featuring significantly more varied layouts and styles. In contrast, our method directly leverages powerful image diffusion models trained on a broad spectrum of text-image pairs and thus retain their exceptional generalization abilities. This objective is realized through the development of an iterative novel view synthesis and geometry alignment algorithm. The algorithm harnesses diffusion models for dual purposes: they provide appearance prior by predicting novel views of the selected object using estimated depth maps, and they act as a geometry critic by correcting misalignments in 3D shapes across the sampled views. Our method can generate high-quality 3D-aware image edits with large viewpoint transformations and high appearance and shape consistency with the input image, pushing the boundaries of what is possible with single-image 3D-aware editing.