ProcGen3D: Learning Neural Procedural Graph Representations for Image-to-3D Reconstruction
This addresses the challenge of efficient and accurate image-to-3D reconstruction for 3D asset creation, particularly in production applications, though it is incremental as it builds on existing procedural generator concepts.
The paper tackles the problem of 3D content creation by generating procedural graph abstractions from images, resulting in a method that outperforms state-of-the-art generative 3D techniques and domain-specific modeling in experiments on objects like cacti, trees, and bridges, with improved generalization to real-world images using only synthetic training data.
We introduce ProcGen3D, a new approach for 3D content creation by generating procedural graph abstractions of 3D objects, which can then be decoded into rich, complex 3D assets. Inspired by the prevalent use of procedural generators in production 3D applications, we propose a sequentialized, graph-based procedural graph representation for 3D assets. We use this to learn to approximate the landscape of a procedural generator for image-based 3D reconstruction. We employ edge-based tokenization to encode the procedural graphs, and train a transformer prior to predict the next token conditioned on an input RGB image. Crucially, to enable better alignment of our generated outputs to an input image, we incorporate Monte Carlo Tree Search (MCTS) guided sampling into our generation process, steering output procedural graphs towards more image-faithful reconstructions. Our approach is applicable across a variety of objects that can be synthesized with procedural generators. Extensive experiments on cacti, trees, and bridges show that our neural procedural graph generation outperforms both state-of-the-art generative 3D methods and domain-specific modeling techniques. Furthermore, this enables improved generalization on real-world input images, despite training only on synthetic data.