SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulation
This work addresses safety in simulation-to-real robot learning, which is crucial for preventing damage in robotics applications, though it is incremental as it builds on existing methods like Bayesian optimization and policy repertoires.
The paper tackles the problem of unsafe policy transfer from simulation to real robots due to the reality gap, and introduces SafeAPT, a learning algorithm that uses a diverse policy repertoire and Bayesian optimization to adapt policies safely, achieving high-performance policies within minutes while minimizing safety violations in real-world experiments.
The framework of Simulation-to-real learning, i.e, learning policies in simulation and transferring those policies to the real world is one of the most promising approaches towards data-efficient learning in robotics. However, due to the inevitable reality gap between the simulation and the real world, a policy learned in the simulation may not always generate a safe behaviour on the real robot. As a result, during adaptation of the policy in the real world, the robot may damage itself or cause harm to its surroundings. In this work, we introduce a novel learning algorithm called SafeAPT that leverages a diverse repertoire of policies evolved in the simulation and transfers the most promising safe policy to the real robot through episodic interaction. To achieve this, SafeAPT iteratively learns a probabilistic reward model as well as a safety model using real-world observations combined with simulated experiences as priors. Then, it performs Bayesian optimization on the repertoire with the reward model while maintaining the specified safety constraint using the safety model. SafeAPT allows a robot to adapt to a wide range of goals safely with the same repertoire of policies evolved in the simulation. We compare SafeAPT with several baselines, both in simulated and real robotic experiments and show that SafeAPT finds high-performance policies within a few minutes in the real world while minimizing safety violations during the interactions.