Learning Action-Effect Dynamics from Pairs of Scene-graphs
This work addresses the challenge of enabling autonomous agents to reason about actions from visual and linguistic inputs, which is incremental as it builds on existing scene-graph representations and datasets.
The paper tackles the problem of reasoning about actions and change by learning action-effect dynamics from pairs of scene-graphs, showing that the proposed approach is effective in terms of performance, data efficiency, and generalization compared to existing models on the CLEVR_HYP dataset.
'Actions' play a vital role in how humans interact with the world. Thus, autonomous agents that would assist us in everyday tasks also require the capability to perform 'Reasoning about Actions & Change' (RAC). Recently, there has been growing interest in the study of RAC with visual and linguistic inputs. Graphs are often used to represent semantic structure of the visual content (i.e. objects, their attributes and relationships among objects), commonly referred to as scene-graphs. In this work, we propose a novel method that leverages scene-graph representation of images to reason about the effects of actions described in natural language. We experiment with existing CLEVR_HYP (Sampat et. al, 2021) dataset and show that our proposed approach is effective in terms of performance, data efficiency, and generalization capability compared to existing models.