BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis
This enables photorealistic novel view synthesis on commodity hardware, with applications in editing and simulation, though it is incremental over existing neural scene representations.
The paper tackles the problem of reconstructing high-quality meshes from neural signed distance fields for real-time view synthesis, achieving improved accuracy, speed, and power consumption compared to previous methods.
We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.