Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning
This work addresses safety for legged robots in hazardous deployments, offering a novel approach to risk-aware locomotion that is incremental by building on existing reinforcement learning methods.
The paper tackled the problem of enabling legged robots to explicitly consider safety risks during locomotion in hazardous environments, proposing a method that uses distributional reinforcement learning to estimate value distributions and integrate risk metrics, resulting in emergent risk-sensitive behaviors demonstrated in simulation and on the ANYmal robot without additional reward tuning.
Deployment in hazardous environments requires robots to understand the risks associated with their actions and movements to prevent accidents. Despite its importance, these risks are not explicitly modeled by currently deployed locomotion controllers for legged robots. In this work, we propose a risk sensitive locomotion training method employing distributional reinforcement learning to consider safety explicitly. Instead of relying on a value expectation, we estimate the complete value distribution to account for uncertainty in the robot's interaction with the environment. The value distribution is consumed by a risk metric to extract risk sensitive value estimates. These are integrated into Proximal Policy Optimization (PPO) to derive our method, Distributional Proximal Policy Optimization (DPPO). The risk preference, ranging from risk-averse to risk-seeking, can be controlled by a single parameter, which enables to adjust the robot's behavior dynamically. Importantly, our approach removes the need for additional reward function tuning to achieve risk sensitivity. We show emergent risk sensitive locomotion behavior in simulation and on the quadrupedal robot ANYmal. Videos of the experiments and code are available at https://sites.google.com/leggedrobotics.com/risk-aware-locomotion.