Extracting Triangular 3D Models, Materials, and Lighting From Images
This addresses the need for deployable 3D models in graphics applications, though it builds incrementally on existing differentiable rendering techniques.
The paper tackles the problem of jointly optimizing topology, materials, and lighting from multi-view images, producing triangle meshes with materials and lighting that can be used in traditional graphics engines. The result enables advanced scene editing, material decomposition, and high-quality view interpolation at interactive rates.
We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers). Project website: https://nvlabs.github.io/nvdiffrec/ .