Action-based image editing guided by human instructions
This addresses the need for more interactive and dynamic image editing tools for users, though it appears incremental by extending static editing to include actions.
The paper tackled the problem of making text-based image editing dynamic by incorporating actions to modify object positions or postures based on human instructions, resulting in substantial improvements in editing and high reasoning capabilities for generating final scenes from starting images.
Text-based image editing is typically approached as a static task that involves operations such as inserting, deleting, or modifying elements of an input image based on human instructions. Given the static nature of this task, in this paper, we aim to make this task dynamic by incorporating actions. By doing this, we intend to modify the positions or postures of objects in the image to depict different actions while maintaining the visual properties of the objects. To implement this challenging task, we propose a new model that is sensitive to action text instructions by learning to recognize contrastive action discrepancies. The model training is done on new datasets defined by extracting frames from videos that show the visual scenes before and after an action. We show substantial improvements in image editing using action-based text instructions and high reasoning capabilities that allow our model to use the input image as a starting scene for an action while generating a new image that shows the final scene of the action.