Adversarial Environment Design via Regret-Guided Diffusion Models
This work addresses the problem of robustness in deep reinforcement learning for agents, offering an incremental improvement in environment generation techniques.
The paper tackles the challenge of training robust reinforcement learning agents by proposing a novel unsupervised environment design method that uses regret-guided diffusion models to generate adversarial training environments, resulting in improved zero-shot generalization across out-of-distribution environments compared to baselines.
Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments. Project page: https://rllab-snu.github.io/projects/ADD