GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images
This addresses the problem of generating realistic free-viewpoint images for human performers, which is important for applications like virtual reality and animation, though it builds incrementally on existing NeRF approaches.
The paper tackles novel view synthesis for arbitrary human performers from sparse multi-view images, proposing GM-NeRF which uses a geometry-guided attention mechanism and neural rendering to outperform state-of-the-art methods on both synthetic and real-world datasets.
In this work, we focus on synthesizing high-fidelity novel view images for arbitrary human performers, given a set of sparse multi-view images. It is a challenging task due to the large variation among articulated body poses and heavy self-occlusions. To alleviate this, we introduce an effective generalizable framework Generalizable Model-based Neural Radiance Fields (GM-NeRF) to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy which can alleviate the misalignment between inaccurate geometry prior and pixel space. On top of that, we further conduct neural rendering and partial gradient backpropagation for efficient perceptual supervision and improvement of the perceptual quality of synthesis. To evaluate our method, we conduct experiments on synthesized datasets THuman2.0 and Multi-garment, and real-world datasets Genebody and ZJUMocap. The results demonstrate that our approach outperforms state-of-the-art methods in terms of novel view synthesis and geometric reconstruction.