Robust Adversarial Policy Optimization Under Dynamics Uncertainty
For RL practitioners, RAPO provides a tractable and effective method for training policies robust to dynamics uncertainty, addressing limitations of existing adversarial and distributionally robust approaches.
The paper addresses the failure of RL policies under dynamics uncertainty and proposes RAPO, a dual formulation that combines trajectory-level adversarial steering and model-level Boltzmann reweighting. RAPO outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics.
Reinforcement learning (RL) policies often fail under dynamics that differ from training, a gap not fully addressed by domain randomization or existing adversarial RL methods. Distributionally robust RL provides a formal remedy but still relies on surrogate adversaries to approximate intractable primal problems, leaving blind spots that potentially cause instability and over-conservatism. We propose a dual formulation that directly exposes the robustness-performance trade-off. At the trajectory level, a temperature parameter from the dual problem is approximated with an adversarial network, yielding efficient and stable worst-case rollouts within a divergence bound. At the model level, we employ Boltzmann reweighting over dynamics ensembles, focusing on more adverse environments to the current policy rather than uniform sampling. The two components act independently and complement each other: trajectory-level steering ensures robust rollouts, while model-level sampling provides policy-sensitive coverage of adverse dynamics. The resulting framework, robust adversarial policy optimization (RAPO) outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics while maintaining dual tractability.