3D Reconstruction from Sketches
This addresses the problem of 3D reconstruction from sketches for computer vision applications, but it is incremental as it combines existing techniques like CycleGAN and MegaDepth.
The paper tackles 3D scene reconstruction from sketches by proposing a pipeline that stitches multiple sketches, converts them to realistic images using CycleGAN, and estimates depth with MegaDepth. The method achieves good performance on single sketches across various drawings, though the stitching component has limited generalization to real drawings.
We consider the problem of reconstructing a 3D scene from multiple sketches. We propose a pipeline which involves (1) stitching together multiple sketches through use of correspondence points, (2) converting the stitched sketch into a realistic image using a CycleGAN, and (3) estimating that image's depth-map using a pre-trained convolutional neural network based architecture called MegaDepth. Our contribution includes constructing a dataset of image-sketch pairs, the images for which are from the Zurich Building Database, and sketches have been generated by us. We use this dataset to train a CycleGAN for our pipeline's second step. We end up with a stitching process that does not generalize well to real drawings, but the rest of the pipeline that creates a 3D reconstruction from a single sketch performs quite well on a wide variety of drawings.