Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)
For researchers in robot learning, this work identifies a fundamental misalignment between sim2real and policy learning, though the proposed solution is preliminary.
The paper argues that excessive sim2real efforts can hinder policy learning by causing simulator lock-in and poor exploration, and proposes a sim2sim2real paradigm that uses robot kinematics as the sole design constraint.
While sim2real efforts are necessary for effective policy transfer to hardware, there is such a thing as too much of a good thing. We argue that sim2real efforts have led to misaligned incentives with policy learning, resulting in simulator lock in and poor policy exploration due to the unreasonable constraints imposed by the real world. We offer a diagnosis and explanation of the current status of the problem, and propose a potential solution via a sim2sim2real paradigm that leverages the robot's kinematics as the sole design constraint.