ZIGNeRF: Zero-shot 3D Scene Representation with Invertible Generative Neural Radiance Fields
This work addresses the problem of 3D scene reconstruction from a single image for applications in computer vision and graphics, representing an incremental advance by extending generative NeRFs with a novel inverter for zero-shot capabilities.
The paper tackles the challenge of creating a 3D scene representation from a single out-of-domain image, introducing ZIGNeRF which performs zero-shot GAN inversion to generate multi-view images and enable 3D operations like rotation and translation, validated on datasets such as Cats, AFHQ, CelebA, CelebA-HQ, and CompCars.
Generative Neural Radiance Fields (NeRFs) have demonstrated remarkable proficiency in synthesizing multi-view images by learning the distribution of a set of unposed images. Despite the aptitude of existing generative NeRFs in generating 3D-consistent high-quality random samples within data distribution, the creation of a 3D representation of a singular input image remains a formidable challenge. In this manuscript, we introduce ZIGNeRF, an innovative model that executes zero-shot Generative Adversarial Network (GAN) inversion for the generation of multi-view images from a single out-of-domain image. The model is underpinned by a novel inverter that maps out-of-domain images into the latent code of the generator manifold. Notably, ZIGNeRF is capable of disentangling the object from the background and executing 3D operations such as 360-degree rotation or depth and horizontal translation. The efficacy of our model is validated using multiple real-image datasets: Cats, AFHQ, CelebA, CelebA-HQ, and CompCars.