Adversarial Diffusion for Robust Reinforcement Learning
This work addresses robustness issues in reinforcement learning, which is crucial for deploying RL in real-world applications with uncertainties, though it appears incremental as it builds on existing connections between CVaR optimization and robust RL.
The paper tackled the challenge of robustness to modeling errors in reinforcement learning by using diffusion models to train policies that are robust to uncertainty in environment dynamics, achieving superior robustness and performance compared to existing methods across standard benchmarks.
Robustness to modeling errors and uncertainties remains a central challenge in reinforcement learning (RL). In this work, we address this challenge by leveraging diffusion models to train robust RL policies. Diffusion models have recently gained popularity in model-based RL due to their ability to generate full trajectories "all at once", mitigating the compounding errors typical of step-by-step transition models. Moreover, they can be conditioned to sample from specific distributions, making them highly flexible. We leverage conditional sampling to learn policies that are robust to uncertainty in environment dynamics. Building on the established connection between Conditional Value at Risk (CVaR) optimization and robust RL, we introduce Adversarial Diffusion for Robust Reinforcement Learning (AD-RRL). AD-RRL guides the diffusion process to generate worst-case trajectories during training, effectively optimizing the CVaR of the cumulative return. Empirical results across standard benchmarks show that AD-RRL achieves superior robustness and performance compared to existing robust RL methods.