Nexels: Neurally-Textured Surfels for Real-Time Novel View Synthesis with Sparse Geometries
This work addresses the computational and memory inefficiencies in real-time novel view synthesis for applications like VR/AR, offering a more compact and faster alternative to existing methods.
The paper tackles the problem of inefficient representation in novel view synthesis by proposing a method that decouples geometry and appearance, achieving compactness with surfels for geometry and neural fields for appearance. It matches the perceptual quality of 3D Gaussian splatting while using significantly fewer primitives (e.g., 9.7x fewer outdoors) and less memory, and renders twice as fast with improved visual quality.
Though Gaussian splatting has achieved impressive results in novel view synthesis, it requires millions of primitives to model highly textured scenes, even when the geometry of the scene is simple. We propose a representation that goes beyond point-based rendering and decouples geometry and appearance in order to achieve a compact representation. We use surfels for geometry and a combination of a global neural field and per-primitive colours for appearance. The neural field textures a fixed number of primitives for each pixel, ensuring that the added compute is low. Our representation matches the perceptual quality of 3D Gaussian splatting while using $9.7\times$ fewer primitives and $5.5\times$ less memory on outdoor scenes and using $31\times$ fewer primitives and $3.7\times$ less memory on indoor scenes. Our representation also renders twice as fast as existing textured primitives while improving upon their visual quality.