NeuTex: Neural Texture Mapping for Volumetric Neural Rendering
This addresses the need for editable neural representations in computer graphics, offering a novel approach for scenes where mesh reconstruction fails, though it is incremental in building on existing volumetric methods.
The paper tackles the problem of entangled geometry and appearance in volumetric neural rendering by proposing NeuTex, which explicitly disentangles geometry as a 3D volume and appearance as a 2D texture map using a texture mapping network with cycle consistency loss, enabling high-quality rendering and user editing of appearance via 2D texture maps.
Recent work has demonstrated that volumetric scene representations combined with differentiable volume rendering can enable photo-realistic rendering for challenging scenes that mesh reconstruction fails on. However, these methods entangle geometry and appearance in a "black-box" volume that cannot be edited. Instead, we present an approach that explicitly disentangles geometry--represented as a continuous 3D volume--from appearance--represented as a continuous 2D texture map. We achieve this by introducing a 3D-to-2D texture mapping (or surface parameterization) network into volumetric representations. We constrain this texture mapping network using an additional 2D-to-3D inverse mapping network and a novel cycle consistency loss to make 3D surface points map to 2D texture points that map back to the original 3D points. We demonstrate that this representation can be reconstructed using only multi-view image supervision and generates high-quality rendering results. More importantly, by separating geometry and texture, we allow users to edit appearance by simply editing 2D texture maps.