3D-aware Conditional Image Synthesis
This work addresses the need for 3D-aware image synthesis with explicit user control, which is incremental by extending existing conditional generative models with neural radiance fields for domain-specific applications like interactive editing.
The paper tackles the problem of controllable photorealistic image synthesis from 2D label maps by proposing pix2pix3D, a 3D-aware conditional generative model that enables rendering images from different viewpoints and allows interactive editing of label maps.
We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available monocular images and label map pairs, our model learns to assign a label to every 3D point in addition to color and density, which enables it to render the image and pixel-aligned label map simultaneously. Finally, we build an interactive system that allows users to edit the label map from any viewpoint and generate outputs accordingly.