Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
This work addresses scene understanding for robotics and simulation by enabling bounce prediction from visual inputs, though it is incremental as it builds on existing physics-based and learning methods.
The paper tackles the problem of modeling surface properties for bounce prediction in everyday scenes by introducing an end-to-end model that predicts post-bounce trajectories and infers physical properties like restitution and collision normals from single images and 3D trajectories. The model outperforms baselines, including Newtonian physics fitting, on a newly collected dataset of 5K RGB-D videos.
We introduce an approach to model surface properties governing bounces in everyday scenes. Our model learns end-to-end, starting from sensor inputs, to predict post-bounce trajectories and infer two underlying physical properties that govern bouncing - restitution and effective collision normals. Our model, Bounce and Learn, comprises two modules -- a Physics Inference Module (PIM) and a Visual Inference Module (VIM). VIM learns to infer physical parameters for locations in a scene given a single still image, while PIM learns to model physical interactions for the prediction task given physical parameters and observed pre-collision 3D trajectories. To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices. Our proposed model learns from our collected dataset of real-world bounces and is bootstrapped with additional information from simple physics simulations. We show on our newly collected dataset that our model out-performs baselines, including trajectory fitting with Newtonian physics, in predicting post-bounce trajectories and inferring physical properties of a scene.