HyperNeRFGAN: Hypernetwork approach to 3D NeRF GAN
This addresses a key bottleneck in 3D object generation for applications like medical imaging where camera positions are hard to estimate, though it appears incremental as it builds on existing NeRF and GAN paradigms.
The paper tackles the problem of training 3D generative models without needing known camera positions, by introducing HyperNeRFGAN, which uses a hypernetwork to generate NeRF weights from noise, resulting in superior performance on diverse image datasets, especially medical data, compared to state-of-the-art alternatives.
The recent surge in popularity of deep generative models for 3D objects has highlighted the need for more efficient training methods, particularly given the difficulties associated with training with conventional 3D representations, such as voxels or point clouds. Neural Radiance Fields (NeRFs), which provide the current benchmark in terms of quality for the generation of novel views of complex 3D scenes from a limited set of 2D images, represent a promising solution to this challenge. However, the training of these models requires the knowledge of the respective camera positions from which the images were viewed. In this paper, we overcome this limitation by introducing HyperNeRFGAN, a Generative Adversarial Network (GAN) architecture employing a hypernetwork paradigm to transform a Gaussian noise into the weights of a NeRF architecture that does not utilize viewing directions in its training phase. Consequently, as evidenced by the findings of our experimental study, the proposed model, despite its notable simplicity in comparison to existing state-of-the-art alternatives, demonstrates superior performance on a diverse range of image datasets where camera position estimation is challenging, particularly in the context of medical data.