Dynamic Scene Novel View Synthesis via Deferred Spatio-temporal Consistency
This addresses the problem of rendering dynamic scenes from sparse inputs for applications like VR or film, but it appears incremental as it builds on existing SfM and NVS techniques.
The paper tackles novel view synthesis for dynamic scenes from casual capture by jointly addressing structure from motion and rendering, using a deferred spatio-temporal consistency approach to handle noisy reconstructions without complex training. It demonstrates results on real-world scenes against baselines, though no specific performance numbers are provided.
Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and reconstruct only a sparse point cloud per timestep that is noisy in space-time. Second, for NVS, we present a variational diffusion formulation on depths and colors that lets us robustly cope with the noise by enforcing spatio-temporal consistency via per-pixel reprojection weights derived from the input views. Together, this deferred approach generates novel views for dynamic scenes without requiring challenging spatio-temporally consistent reconstructions nor training complex models on large datasets. We demonstrate our algorithm on real-world dynamic scenes against classic and more recent learning-based baseline approaches.