Voxel-Mesh Hybrid Representation for Real-Time View Synthesis
This work addresses the problem of real-time, high-quality view synthesis for applications like VR/AR, though it is incremental as it builds on existing NeRF methods by combining voxel and mesh components.
The paper tackles the trade-off between rendering quality and speed in neural radiance fields (NeRF) by proposing Vosh, a hybrid voxel-mesh representation that enables real-time view synthesis on mobile devices, achieving a commendable balance with interactive performance.
The neural radiance fields (NeRF) have emerged as a prominent methodology for synthesizing realistic images of novel views. While neural radiance representations based on voxels or mesh individually offer distinct advantages, excelling in either rendering quality or speed, each has limitations in the other aspect. In response, we propose a hybrid representation named Vosh, seamlessly combining both voxel and mesh components in hybrid rendering for view synthesis. Vosh is meticulously crafted by optimizing the voxel grid based on neural rendering, strategically meshing a portion of the volumetric density field to surface. Therefore, it excels in fast rendering scenes with simple geometry and textures through its mesh component, while simultaneously enabling high-quality rendering in intricate regions by leveraging voxel component. The flexibility of Vosh is showcased through the ability to adjust hybrid ratios, providing users the ability to control the balance between rendering quality and speed based on flexible usage. Experimental results demonstrate that our method achieves commendable trade-off between rendering quality and speed, and notably has real-time performance on mobile devices. The interactive web demo and code are available at https://zyyzyy06.github.io/Vosh.