WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild
This addresses the problem of limited multi-view training data for scene-level novel view synthesis, enabling applications with appearance control, though it is incremental as it extends an existing multi-view diffusion paradigm.
The paper tackles the challenge of scene-level novel view synthesis by training on diverse, permissively-licensed 2D scene images from the wild, achieving state-of-the-art results in single-view novel view synthesis for both object- and scene-level settings with less data than prior methods.
Despite recent advances in sparse novel view synthesis (NVS) applied to object-centric scenes, scene-level NVS remains a challenge. A central issue is the lack of available clean multi-view training data, beyond manually curated datasets with limited diversity, camera variation, or licensing issues. On the other hand, an abundance of diverse and permissively-licensed data exists in the wild, consisting of scenes with varying appearances (illuminations, transient occlusions, etc.) from sources such as tourist photos. To this end, we present WildCAT3D, a framework for generating novel views of scenes learned from diverse 2D scene image data captured in the wild. We unlock training on these data sources by explicitly modeling global appearance conditions in images, extending the state-of-the-art multi-view diffusion paradigm to learn from scene views of varying appearances. Our trained model generalizes to new scenes at inference time, enabling the generation of multiple consistent novel views. WildCAT3D provides state-of-the-art results on single-view NVS in object- and scene-level settings, while training on strictly less data sources than prior methods. Additionally, it enables novel applications by providing global appearance control during generation.