Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation
This addresses the challenge for computer vision and graphics researchers in creating high-quality 3D-consistent images, representing an incremental improvement by combining 3D rendering and 2D super-resolution strategies.
The paper tackles the problem of generating photorealistic and multiview 3D-consistent images in 3D-aware GANs, achieving FID scores of 5.4 on FFHQ and 4.3 on AFHQ-v2 Cats at 512x512 resolution, which outperforms direct 3D rendering methods and approaches state-of-the-art 2D super-resolution techniques.
Generating images with both photorealism and multiview 3D consistency is crucial for 3D-aware GANs, yet existing methods struggle to achieve them simultaneously. Improving the photorealism via CNN-based 2D super-resolution can break the strict 3D consistency, while keeping the 3D consistency by learning high-resolution 3D representations for direct rendering often compromises image quality. In this paper, we propose a novel learning strategy, namely 3D-to-2D imitation, which enables a 3D-aware GAN to generate high-quality images while maintaining their strict 3D consistency, by letting the images synthesized by the generator's 3D rendering branch to mimic those generated by its 2D super-resolution branch. We also introduce 3D-aware convolutions into the generator for better 3D representation learning, which further improves the image generation quality. With the above strategies, our method reaches FID scores of 5.4 and 4.3 on FFHQ and AFHQ-v2 Cats, respectively, at 512x512 resolution, largely outperforming existing 3D-aware GANs using direct 3D rendering and coming very close to the previous state-of-the-art method that leverages 2D super-resolution. Project website: https://seanchenxy.github.io/Mimic3DWeb.