Deep Sketch-Based Modeling: Tips and Tricks
This work addresses the challenge of creating 3D models from sketches for designers and artists, but it is incremental as it adapts existing methods rather than introducing a new paradigm.
The paper tackles the problem of sketch-based 3D modeling by identifying key differences from image-based inputs—style variance, imprecise perspective, and sparsity—and shows how adapting deep single-image modeling methods to address these differences improves performance.
Deep image-based modeling received lots of attention in recent years, yet the parallel problem of sketch-based modeling has only been briefly studied, often as a potential application. In this work, for the first time, we identify the main differences between sketch and image inputs: (i) style variance, (ii) imprecise perspective, and (iii) sparsity. We discuss why each of these differences can pose a challenge, and even make a certain class of image-based methods inapplicable. We study alternative solutions to address each of the difference. By doing so, we drive out a few important insights: (i) sparsity commonly results in an incorrect prediction of foreground versus background, (ii) diversity of human styles, if not taken into account, can lead to very poor generalization properties, and finally (iii) unless a dedicated sketching interface is used, one can not expect sketches to match a perspective of a fixed viewpoint. Finally, we compare a set of representative deep single-image modeling solutions and show how their performance can be improved to tackle sketch input by taking into consideration the identified critical differences.