Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpoint
This addresses the sim-to-real gap problem for robotics control, offering a method to transfer and adapt skills from simulation to real-world environments, though it is incremental as it builds on existing representation learning techniques.
The paper tackles sim-to-real skill transfer and discovery for robotics control by using representation learning inspired by spectral decomposition, enabling transfer of skills learned in simulation to multiple real-world tasks like hovering and trajectory tracking. The result is a 30.2% improvement in real-world controller performance by discovering new skills to handle the sim-to-real gap.
We study sim-to-real skill transfer and discovery in the context of robotics control using representation learning. We draw inspiration from spectral decomposition of Markov decision processes. The spectral decomposition brings about representation that can linearly represent the state-action value function induced by any policies, thus can be regarded as skills. The skill representations are transferable across arbitrary tasks with the same transition dynamics. Moreover, to handle the sim-to-real gap in the dynamics, we propose a skill discovery algorithm that learns new skills caused by the sim-to-real gap from real-world data. We promote the discovery of new skills by enforcing orthogonal constraints between the skills to learn and the skills from simulators, and then synthesize the policy using the enlarged skill sets. We demonstrate our methodology by transferring quadrotor controllers from simulators to Crazyflie 2.1 quadrotors. We show that we can learn the skill representations from a single simulator task and transfer these to multiple different real-world tasks including hovering, taking off, landing and trajectory tracking. Our skill discovery approach helps narrow the sim-to-real gap and improve the real-world controller performance by up to 30.2%.