Text as Neural Operator: Image Manipulation by Text Instruction
This addresses a multimedia and computer vision problem for users needing precise image editing via text, but it is incremental as it builds on existing text-guided manipulation methods.
The paper tackles the problem of editing images with multiple objects using complex text instructions to add, remove, or change objects, proposing a GAN-based method that treats text as neural operators to locally modify image features, and it shows favorable performance against baselines on three datasets with greater fidelity and semantic relevance.
In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community. The input to conditional image generation has evolved from image-only to multimodality. In this paper, we study a setting that allows users to edit an image with multiple objects using complex text instructions to add, remove, or change the objects. The inputs of the task are multimodal including (1) a reference image and (2) an instruction in natural language that describes desired modifications to the image. We propose a GAN-based method to tackle this problem. The key idea is to treat text as neural operators to locally modify the image feature. We show that the proposed model performs favorably against recent strong baselines on three public datasets. Specifically, it generates images of greater fidelity and semantic relevance, and when used as a image query, leads to better retrieval performance.