Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
This work addresses safety concerns for offline RL applications, but it is incremental as it builds on existing risk-averse methods by incorporating environmental stochasticity.
The paper tackles the challenge of ensuring safety in offline reinforcement learning by addressing both epistemic uncertainty and environmental stochasticity, proposing a model-free algorithm that learns risk-averse policies and characterizes the entire reward distribution, with experiments showing superior performance in benchmarks.
Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties associated with various actions and environmental stochasticity. Traditional approaches primarily emphasize mitigating epistemic uncertainty by learning risk-averse policies, often overlooking environmental stochasticity. In this study, we propose an uncertainty-aware distributional offline RL method to simultaneously address both epistemic uncertainty and environmental stochasticity. We propose a model-free offline RL algorithm capable of learning risk-averse policies and characterizing the entire distribution of discounted cumulative rewards, as opposed to merely maximizing the expected value of accumulated discounted returns. Our method is rigorously evaluated through comprehensive experiments in both risk-sensitive and risk-neutral benchmarks, demonstrating its superior performance.