End-to-End Visual Editing with a Generatively Pre-Trained Artist
This addresses the image editing problem for users needing precise blending, though it is incremental as it builds on existing auto-regressive transformer methods.
The paper tackles the targeted image editing problem by learning a conditional probability distribution of edits end-to-end, using a self-supervised approach that simulates edits from off-the-shelf images. It demonstrates significantly better edits and efficiency, outperforming prior work in quantitative and qualitative experiments.
We consider the targeted image editing problem: blending a region in a source image with a driver image that specifies the desired change. Differently from prior works, we solve this problem by learning a conditional probability distribution of the edits, end-to-end. Training such a model requires addressing a fundamental technical challenge: the lack of example edits for training. To this end, we propose a self-supervised approach that simulates edits by augmenting off-the-shelf images in a target domain. The benefits are remarkable: implemented as a state-of-the-art auto-regressive transformer, our approach is simple, sidesteps difficulties with previous methods based on GAN-like priors, obtains significantly better edits, and is efficient. Furthermore, we show that different blending effects can be learned by an intuitive control of the augmentation process, with no other changes required to the model architecture. We demonstrate the superiority of this approach across several datasets in extensive quantitative and qualitative experiments, including human studies, significantly outperforming prior work.