Temporal-MPI: Enabling Multi-Plane Images for Dynamic Scene Modelling via Temporal Basis Learning
This addresses the challenge of immersive rendering for dynamic scenes, which is incremental as it extends the multi-plane image method from static to dynamic contexts.
The paper tackles the problem of novel view synthesis for dynamic scenes by proposing Temporal-MPI, a representation that encodes 3D and dynamic variations via learned temporal basis and coefficients, enabling real-time rendering with high visual quality. It demonstrates that Temporal-MPI is faster and more compact than state-of-the-art methods on the Nvidia Dynamic Scene Dataset.
Novel view synthesis of static scenes has achieved remarkable advancements in producing photo-realistic results. However, key challenges remain for immersive rendering of dynamic scenes. One of the seminal image-based rendering method, the multi-plane image (MPI), produces high novel-view synthesis quality for static scenes. But modelling dynamic contents by MPI is not studied. In this paper, we propose a novel Temporal-MPI representation which is able to encode the rich 3D and dynamic variation information throughout the entire video as compact temporal basis and coefficients jointly learned. Time-instance MPI for rendering can be generated efficiently using mini-seconds by linear combinations of temporal basis and coefficients from Temporal-MPI. Thus novel-views at arbitrary time-instance will be able to be rendered via Temporal-MPI in real-time with high visual quality. Our method is trained and evaluated on Nvidia Dynamic Scene Dataset. We show that our proposed Temporal- MPI is much faster and more compact compared with other state-of-the-art dynamic scene modelling methods.