Cross-Modal 3D Shape Generation and Manipulation
This addresses the challenge of making 3D design more accessible and intuitive for users by leveraging natural 2D interactions, though it appears incremental as it builds on existing multi-modal and implicit representation methods.
The paper tackles the problem of creating and editing 3D objects, which is labor-intensive, by proposing a multi-modal generative model that uses 2D inputs like sketches and images to generate and manipulate 3D shapes and colors, enabling tasks such as shape editing via sketches and recoloring via scribbles without per-task retraining.
Creating and editing the shape and color of 3D objects require tremendous human effort and expertise. Compared to direct manipulation in 3D interfaces, 2D interactions such as sketches and scribbles are usually much more natural and intuitive for the users. In this paper, we propose a generic multi-modal generative model that couples the 2D modalities and implicit 3D representations through shared latent spaces. With the proposed model, versatile 3D generation and manipulation are enabled by simply propagating the editing from a specific 2D controlling modality through the latent spaces. For example, editing the 3D shape by drawing a sketch, re-colorizing the 3D surface via painting color scribbles on the 2D rendering, or generating 3D shapes of a certain category given one or a few reference images. Unlike prior works, our model does not require re-training or fine-tuning per editing task and is also conceptually simple, easy to implement, robust to input domain shifts, and flexible to diverse reconstruction on partial 2D inputs. We evaluate our framework on two representative 2D modalities of grayscale line sketches and rendered color images, and demonstrate that our method enables various shape manipulation and generation tasks with these 2D modalities.