Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization
This addresses safety concerns for RL in real-world applications where policies must withstand perturbations, though it appears incremental as it builds on existing DRL algorithms like TD3 and PPO.
The paper tackles the vulnerability of reinforcement learning policies to action perturbations by introducing Optimal Adversary-aware Policy Iteration (OA-PI), which improves robustness while maintaining nominal performance and sample efficiency in various environments.
Reinforcement Learning (RL) has achieved remarkable success in sequential decision tasks. However, recent studies have revealed the vulnerability of RL policies to different perturbations, raising concerns about their effectiveness and safety in real-world applications. In this work, we focus on the robustness of RL policies against action perturbations and introduce a novel framework called Optimal Adversary-aware Policy Iteration (OA-PI). Our framework enhances action robustness under various perturbations by evaluating and improving policy performance against the corresponding optimal adversaries. Besides, our approach can be integrated into mainstream DRL algorithms such as Twin Delayed DDPG (TD3) and Proximal Policy Optimization (PPO), improving action robustness effectively while maintaining nominal performance and sample efficiency. Experimental results across various environments demonstrate that our method enhances robustness of DRL policies against different action adversaries effectively.