Patch-based 3D Natural Scene Generation from a Single Example
This addresses the challenge of 3D scene generation for natural environments where training data is scarce, though it appears incremental as an extension of classical 2D patch-based methods to 3D.
The paper tackles the problem of generating 3D natural scenes from a single example by proposing a patch-based generative model, which produces high-quality scenes with realistic geometry and appearance in large quantities and varieties.
We target a 3D generative model for general natural scenes that are typically unique and intricate. Lacking the necessary volumes of training data, along with the difficulties of having ad hoc designs in presence of varying scene characteristics, renders existing setups intractable. Inspired by classical patch-based image models, we advocate for synthesizing 3D scenes at the patch level, given a single example. At the core of this work lies important algorithmic designs w.r.t the scene representation and generative patch nearest-neighbor module, that address unique challenges arising from lifting classical 2D patch-based framework to 3D generation. These design choices, on a collective level, contribute to a robust, effective, and efficient model that can generate high-quality general natural scenes with both realistic geometric structure and visual appearance, in large quantities and varieties, as demonstrated upon a variety of exemplar scenes.