GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation
This work improves 3D-aware image generation for applications like computer graphics and virtual reality, representing an incremental advance over prior methods.
The paper tackles the problem of generating 3D-consistent images with fine details by addressing the high memory and computation costs in neural radiance field generators, resulting in high-quality images with realistic details and strong 3D consistency.
3D-aware image generative modeling aims to generate 3D-consistent images with explicitly controllable camera poses. Recent works have shown promising results by training neural radiance field (NeRF) generators on unstructured 2D images, but still can not generate highly-realistic images with fine details. A critical reason is that the high memory and computation cost of volumetric representation learning greatly restricts the number of point samples for radiance integration during training. Deficient sampling not only limits the expressive power of the generator to handle fine details but also impedes effective GAN training due to the noise caused by unstable Monte Carlo sampling. We propose a novel approach that regulates point sampling and radiance field learning on 2D manifolds, embodied as a set of learned implicit surfaces in the 3D volume. For each viewing ray, we calculate ray-surface intersections and accumulate their radiance generated by the network. By training and rendering such radiance manifolds, our generator can produce high quality images with realistic fine details and strong visual 3D consistency.