Learning Visibility Field for Detailed 3D Human Reconstruction and Relighting
This work addresses the challenge of creating realistic digital humans for applications like VR and film, offering an incremental improvement by enhancing existing methods with a visibility field for better occlusion handling and relighting.
The paper tackles the problem of detailed 3D human reconstruction and relighting from sparse views by proposing a framework that integrates occupancy, albedo, and a novel visibility field to resolve occlusion ambiguity and enable self-shadowed relighting, achieving state-of-the-art reconstruction accuracy and comparable relighting accuracy to ray-traced ground truth.
Detailed 3D reconstruction and photo-realistic relighting of digital humans are essential for various applications. To this end, we propose a novel sparse-view 3d human reconstruction framework that closely incorporates the occupancy field and albedo field with an additional visibility field--it not only resolves occlusion ambiguity in multiview feature aggregation, but can also be used to evaluate light attenuation for self-shadowed relighting. To enhance its training viability and efficiency, we discretize visibility onto a fixed set of sample directions and supply it with coupled geometric 3D depth feature and local 2D image feature. We further propose a novel rendering-inspired loss, namely TransferLoss, to implicitly enforce the alignment between visibility and occupancy field, enabling end-to-end joint training. Results and extensive experiments demonstrate the effectiveness of the proposed method, as it surpasses state-of-the-art in terms of reconstruction accuracy while achieving comparably accurate relighting to ray-traced ground truth.