Exploring the GLIDE model for Human Action-effect Prediction
This addresses the action-effect prediction task for computer vision applications, but it is incremental as it applies an existing method to a new task.
The paper tackles the problem of predicting the visual effect of an action in an image by using the GLIDE model to inpaint masked regions based on text descriptions, with qualitative results demonstrated on the EPIC dataset.
We address the following action-effect prediction task. Given an image depicting an initial state of the world and an action expressed in text, predict an image depicting the state of the world following the action. The prediction should have the same scene context as the input image. We explore the use of the recently proposed GLIDE model for performing this task. GLIDE is a generative neural network that can synthesize (inpaint) masked areas of an image, conditioned on a short piece of text. Our idea is to mask-out a region of the input image where the effect of the action is expected to occur. GLIDE is then used to inpaint the masked region conditioned on the required action. In this way, the resulting image has the same background context as the input image, updated to show the effect of the action. We give qualitative results from experiments using the EPIC dataset of ego-centric videos labelled with actions.