VMesh: Hybrid Volume-Mesh Representation for Efficient View Synthesis
This addresses the challenge of combining high-quality view synthesis with real-time performance and editability for immersive applications, representing an incremental improvement over existing methods.
The paper tackles the problem of high rendering costs and limited editability in neural radiance fields (NeRFs) by introducing VMesh, a hybrid volume-mesh representation that enables efficient view synthesis, achieving 2K 60FPS rendering on consumer devices.
With the emergence of neural radiance fields (NeRFs), view synthesis quality has reached an unprecedented level. Compared to traditional mesh-based assets, this volumetric representation is more powerful in expressing scene geometry but inevitably suffers from high rendering costs and can hardly be involved in further processes like editing, posing significant difficulties in combination with the existing graphics pipeline. In this paper, we present a hybrid volume-mesh representation, VMesh, which depicts an object with a textured mesh along with an auxiliary sparse volume. VMesh retains the advantages of mesh-based assets, such as efficient rendering, compact storage, and easy editing, while also incorporating the ability to represent subtle geometric structures provided by the volumetric counterpart. VMesh can be obtained from multi-view images of an object and renders at 2K 60FPS on common consumer devices with high fidelity, unleashing new opportunities for real-time immersive applications.