Baking Relightable NeRF for Real-time Direct/Indirect Illumination Rendering
This addresses the problem of high computation costs in real-time relighting for applications like virtual reality or gaming, though it appears incremental by building on existing neural rendering techniques.
The paper tackles the challenge of real-time relighting for immersive photo-realistic experiences by proposing a method that uses a CNN renderer for direct illumination and a lightweight hash grid-based renderer for indirect illumination, achieving real-time physically-based rendering under unseen lighting conditions with negligible quality loss.
Relighting, which synthesizes a novel view under a given lighting condition (unseen in training time), is a must feature for immersive photo-realistic experience. However, real-time relighting is challenging due to high computation cost of the rendering equation which requires shape and material decomposition and visibility test to model shadow. Additionally, for indirect illumination, additional computation of rendering equation on each secondary surface point (where reflection occurs) is required rendering real-time relighting challenging. We propose a novel method that executes a CNN renderer to compute primary surface points and rendering parameters, required for direct illumination. We also present a lightweight hash grid-based renderer, for indirect illumination, which is recursively executed to perform the secondary ray tracing process. Both renderers are trained in a distillation from a pre-trained teacher model and provide real-time physically-based rendering under unseen lighting condition at a negligible loss of rendering quality.