Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation
This addresses the challenge of robust policy learning in robotics for tasks requiring adaptation to real-world variations, though it is incremental by building on existing imitation and reinforcement learning methods.
The paper tackles the problem of learning robust robotic manipulation policies without extensive human supervision or unsafe real-world data collection by proposing RialTo, a system that uses a real-to-sim-to-real approach with digital twins and inverse distillation, resulting in over 67% increase in policy robustness across tasks like stacking dishes and placing books.
Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/