ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
This addresses the challenge of realistic image compositing for applications like visualizing furniture in rooms or accessories on portraits, representing an incremental improvement with a hybrid method.
The paper tackles the problem of realistically compositing foreground objects into background images by proposing ST-GAN, a novel GAN architecture using Spatial Transformer Networks as the generator, which achieves better results through an iterative warping scheme and sequential training strategy.
We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image. To achieve this, we propose a novel Generative Adversarial Network (GAN) architecture that utilizes Spatial Transformer Networks (STNs) as the generator, which we call Spatial Transformer GANs (ST-GANs). ST-GANs seek image realism by operating in the geometric warp parameter space. In particular, we exploit an iterative STN warping scheme and propose a sequential training strategy that achieves better results compared to naive training of a single generator. One of the key advantages of ST-GAN is its applicability to high-resolution images indirectly since the predicted warp parameters are transferable between reference frames. We demonstrate our approach in two applications: (1) visualizing how indoor furniture (e.g. from product images) might be perceived in a room, (2) hallucinating how accessories like glasses would look when matched with real portraits.