DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images
This addresses the challenge of making image editing more controllable and explainable for users relying on natural language instructions, though it appears incremental as it builds on existing text-based editing methods.
The paper tackles the problem of text-based semantic image editing by proposing DM-Align, a model that uses word alignments to explicitly reason about which parts of an image to alter or preserve, resulting in superior performance in preserving backgrounds and handling long text instructions compared to state-of-the-art baselines.
Text-based semantic image editing assumes the manipulation of an image using a natural language instruction. Although recent works are capable of generating creative and qualitative images, the problem is still mostly approached as a black box sensitive to generating unexpected outputs. Therefore, we propose a novel model to enhance the text-based control of an image editor by explicitly reasoning about which parts of the image to alter or preserve. It relies on word alignments between a description of the original source image and the instruction that reflects the needed updates, and the input image. The proposed Diffusion Masking with word Alignments (DM-Align) allows the editing of an image in a transparent and explainable way. It is evaluated on a subset of the Bison dataset and a self-defined dataset dubbed Dream. When comparing to state-of-the-art baselines, quantitative and qualitative results show that DM-Align has superior performance in image editing conditioned on language instructions, well preserves the background of the image and can better cope with long text instructions.