IterGANs: Iterative GANs to Learn and Control 3D Object Transformation
This addresses the challenge of 3D object manipulation from 2D images for computer vision applications, but it appears incremental as it builds on existing GAN frameworks.
The paper tackles the problem of learning visual representations for 3D manipulations from a single 2D image by proposing Iterative Generative Adversarial Networks (IterGANs), which iteratively transform images and learn implicit 3D and appearance models, showing effectiveness in experiments on rotated objects and scenes.
We are interested in learning visual representations which allow for 3D manipulations of visual objects based on a single 2D image. We cast this into an image-to-image transformation task, and propose Iterative Generative Adversarial Networks (IterGANs) which iteratively transform an input image into an output image. Our models learn a visual representation that can be used for objects seen in training, but also for never seen objects. Since object manipulation requires a full understanding of the geometry and appearance of the object, our IterGANs learn an implicit 3D model and a full appearance model of the object, which are both inferred from a single (test) image. Two advantages of IterGANs are that the intermediate generated images can be used for an additional supervision signal, even in an unsupervised fashion, and that the number of iterations can be used as a control signal to steer the transformation. Experiments on rotated objects and scenes show how IterGANs help with the generation process.