SVS: Adversarial refinement for sparse novel view synthesis
This addresses a critical bottleneck in 3D reconstruction and view synthesis for applications like VR/AR, though it is an incremental improvement over existing radiance field approaches.
The paper tackles the problem of sparse novel view synthesis, where radiance field methods fail due to artifacts from limited reference views and large baselines, by unifying radiance fields with adversarial learning and perceptual losses to hallucinate plausible scene contents, achieving up to 60% improvement in perceptual accuracy over state-of-the-art methods.
This paper proposes Sparse View Synthesis. This is a view synthesis problem where the number of reference views is limited, and the baseline between target and reference view is significant. Under these conditions, current radiance field methods fail catastrophically due to inescapable artifacts such 3D floating blobs, blurring and structural duplication, whenever the number of reference views is limited, or the target view diverges significantly from the reference views. Advances in network architecture and loss regularisation are unable to satisfactorily remove these artifacts. The occlusions within the scene ensure that the true contents of these regions is simply not available to the model. In this work, we instead focus on hallucinating plausible scene contents within such regions. To this end we unify radiance field models with adversarial learning and perceptual losses. The resulting system provides up to 60% improvement in perceptual accuracy compared to current state-of-the-art radiance field models on this problem.