NeX: Real-time View Synthesis with Neural Basis Expansion
This addresses the problem of real-time, high-quality view synthesis for applications like VR/AR, with incremental improvements in speed and detail.
The paper tackles novel view synthesis by enhancing multiplane images with a neural basis expansion to model view-dependent effects, achieving state-of-the-art results on benchmarks and rendering over 1000 times faster than previous methods.
We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. Unlike traditional MPI that uses a set of simple RGB$α$ planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000$\times$ faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/