A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation
This work addresses the real-to-sim problem for robotics researchers working on deformable object manipulation, offering a probabilistic approach to improve state representation.
The paper tackles the challenge of inferring physical parameters for deformable objects like cloth and ropes by framing it as a probabilistic inference task using a simulator, and proposes a method that estimates posterior distributions of properties such as elasticity, friction, and scale from image sequences.
Deformable object manipulation remains a challenging task in robotics research. Conventional techniques for parameter inference and state estimation typically rely on a precise definition of the state space and its dynamics. While this is appropriate for rigid objects and robot states, it is challenging to define the state space of a deformable object and how it evolves in time. In this work, we pose the problem of inferring physical parameters of deformable objects as a probabilistic inference task defined with a simulator. We propose a novel methodology for extracting state information from image sequences via a technique to represent the state of a deformable object as a distribution embedding. This allows to incorporate noisy state observations directly into modern Bayesian simulation-based inference tools in a principled manner. Our experiments confirm that we can estimate posterior distributions of physical properties, such as elasticity, friction and scale of highly deformable objects, such as cloth and ropes. Overall, our method addresses the real-to-sim problem probabilistically and helps to better represent the evolution of the state of deformable objects.