LookinGood^π: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture
This addresses the need for high-quality, real-time human performance capture for applications like virtual reality or film, though it appears incremental as it builds on existing neural rendering techniques.
The paper tackles the problem of low-quality rendering from human performance capture systems by introducing LookinGood^π, a neural re-rendering approach that enhances image quality in real-time and generalizes to unseen people, outperforming state-of-the-art methods.
We propose LookinGood^π, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts (i.e. holes, noise) and obtain a coarse version of the rendered input, while a detail branch is designed to predict "correct" details from the warped references. The guidance of the rendered image is realized by blending features from two branches effectively in the training of the detail branch, which improves both the warping accuracy and the details' fidelity. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.