Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion
This addresses safety-critical constraints for real-world robotic applications, though it appears incremental as it builds upon existing constrained RL methods.
The authors tackled the problem of ensuring safety constraints in deep reinforcement learning for real-world quadrupedal robot locomotion by introducing Guided Constrained Policy Optimization (GCPO), which demonstrated faster convergence to optimal and physically feasible control behavior compared to unconstrained methods.
Deep reinforcement learning (RL) uses model-free techniques to optimize task-specific control policies. Despite having emerged as a promising approach for complex problems, RL is still hard to use reliably for real-world applications. Apart from challenges such as precise reward function tuning, inaccurate sensing and actuation, and non-deterministic response, existing RL methods do not guarantee behavior within required safety constraints that are crucial for real robot scenarios. In this regard, we introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained proximal policy optimization (CPPO) for tracking base velocity commands while following the defined constraints. We also introduce schemes which encourage state recovery into constrained regions in case of constraint violations. We present experimental results of our training method and test it on the real ANYmal quadruped robot. We compare our approach against the unconstrained RL method and show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.