Worst Cases Policy Gradients
This addresses safety concerns in reinforcement learning for domains like autonomous driving, though it is incremental as it builds on existing actor-critic methods with risk modeling.
The paper tackles the problem of learning safe control policies for safety-critical applications by proposing an actor-critic framework that models uncertainty and optimizes policies based on conditional Value-at-Risk, enabling dynamic risk-sensitive actions. The result shows that risk-averse policies generalize better in driving simulations, with significant improvements over other reinforcement learning approaches when tested with different simulation parameters.
Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments. When learning policies for safety-critical applications, it is essential to be sensitive to risks and avoid catastrophic events. Towards this goal, we propose an actor-critic framework that models the uncertainty of the future and simultaneously learns a policy based on that uncertainty model. Specifically, given a distribution of the future return for any state and action, we optimize policies for varying levels of conditional Value-at-Risk. The learned policy can map the same state to different actions depending on the propensity for risk. We demonstrate the effectiveness of our approach in the domain of driving simulations, where we learn maneuvers in two scenarios. Our learned controller can dynamically select actions along a continuous axis, where safe and conservative behaviors are found at one end while riskier behaviors are found at the other. Finally, when testing with very different simulation parameters, our risk-averse policies generalize significantly better compared to other reinforcement learning approaches.