SimVS: Simulating World Inconsistencies for Robust View Synthesis
This addresses the challenge of robust view synthesis for real-world applications by handling scene inconsistencies, though it appears incremental as it builds on existing multi-view datasets and generative models.
The paper tackles the problem of novel-view synthesis in casual capture settings with inconsistencies like varying illumination and scene motion, and demonstrates that their world-simulation strategy significantly outperforms traditional augmentation methods, enabling highly accurate static 3D reconstructions.
Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs