Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning
This addresses safety concerns for RL agents in high-risk domains, though it represents an incremental improvement over existing uncertainty-aware methods.
The paper tackles the problem of safe reinforcement learning in high-risk environments by developing a model-based algorithm that uses bootstrap ensemble uncertainty estimates to guide planning toward low-uncertainty states, resulting in reduced reward attainment as a trade-off for risk sensitivity.
Identifying uncertainty and taking mitigating actions is crucial for safe and trustworthy reinforcement learning agents, especially when deployed in high-risk environments. In this paper, risk sensitivity is promoted in a model-based reinforcement learning algorithm by exploiting the ability of a bootstrap ensemble of dynamics models to estimate environment epistemic uncertainty. We propose uncertainty guided cross-entropy method planning, which penalises action sequences that result in high variance state predictions during model rollouts, guiding the agent to known areas of the state space with low uncertainty. Experiments display the ability for the agent to identify uncertain regions of the state space during planning and to take actions that maintain the agent within high confidence areas, without the requirement of explicit constraints. The result is a reduction in the performance in terms of attaining reward, displaying a trade-off between risk and return.