Re-ReND: Real-time Rendering of NeRFs across Devices
This enables real-time NeRF rendering on mobiles and AR/VR headsets, addressing a bottleneck for practical deployment, but it is incremental as it builds on existing NeRF distillation techniques.
This paper tackles the problem of rendering pre-trained Neural Radiance Fields (NeRFs) in real-time on resource-constrained devices by proposing Re-ReND, which converts NeRFs into a mesh and factorized light field representation, achieving over a 2.6-fold increase in rendering speed compared to state-of-the-art methods without perceptible quality loss.
This paper proposes a novel approach for rendering a pre-trained Neural Radiance Field (NeRF) in real-time on resource-constrained devices. We introduce Re-ReND, a method enabling Real-time Rendering of NeRFs across Devices. Re-ReND is designed to achieve real-time performance by converting the NeRF into a representation that can be efficiently processed by standard graphics pipelines. The proposed method distills the NeRF by extracting the learned density into a mesh, while the learned color information is factorized into a set of matrices that represent the scene's light field. Factorization implies the field is queried via inexpensive MLP-free matrix multiplications, while using a light field allows rendering a pixel by querying the field a single time-as opposed to hundreds of queries when employing a radiance field. Since the proposed representation can be implemented using a fragment shader, it can be directly integrated with standard rasterization frameworks. Our flexible implementation can render a NeRF in real-time with low memory requirements and on a wide range of resource-constrained devices, including mobiles and AR/VR headsets. Notably, we find that Re-ReND can achieve over a 2.6-fold increase in rendering speed versus the state-of-the-art without perceptible losses in quality.