Learning to Generate Chairs, Tables and Cars with Convolutional Networks
This work addresses the problem of generating and manipulating 3D object images for computer vision and graphics applications, representing a novel method for a known bottleneck.
The authors trained generative up-convolutional neural networks to create images of chairs, tables, and cars based on object style, viewpoint, and color, using rendered 3D models. The networks learned meaningful 3D representations, enabling tasks like view interpolation, extrapolation, inventing new objects by recombining training instances, and outperforming existing methods in finding correspondences between objects.
We train generative 'up-convolutional' neural networks which are able to generate images of objects given object style, viewpoint, and color. We train the networks on rendered 3D models of chairs, tables, and cars. Our experiments show that the networks do not merely learn all images by heart, but rather find a meaningful representation of 3D models allowing them to assess the similarity of different models, interpolate between given views to generate the missing ones, extrapolate views, and invent new objects not present in the training set by recombining training instances, or even two different object classes. Moreover, we show that such generative networks can be used to find correspondences between different objects from the dataset, outperforming existing approaches on this task.