Visual Reaction: Learning to Play Catch with Your Drone
This addresses the challenge of real-time interaction in dynamic visual environments for robotics applications, though it is incremental as it builds on existing forecasting and planning methods.
The paper tackles the problem of visual reaction, where an agent interacts with dynamic environments not caused by itself, by developing a model that integrates a forecaster with a planner to play catch with a drone in synthetic environments, outperforming strong baselines.
In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself. Visual reaction entails predicting the future changes in a visual environment and planning accordingly. We study the problem of visual reaction in the context of playing catch with a drone in visually rich synthetic environments. This is a challenging problem since the agent is required to learn (1) how objects with different physical properties and shapes move, (2) what sequence of actions should be taken according to the prediction, (3) how to adjust the actions based on the visual feedback from the dynamic environment (e.g., when objects bouncing off a wall), and (4) how to reason and act with an unexpected state change in a timely manner. We propose a new dataset for this task, which includes 30K throws of 20 types of objects in different directions with different forces. Our results show that our model that integrates a forecaster with a planner outperforms a set of strong baselines that are based on tracking as well as pure model-based and model-free RL baselines. The code and dataset are available at github.com/KuoHaoZeng/Visual_Reaction.