Attacking and Defending Deep Reinforcement Learning Policies
This work addresses security concerns for safety-critical applications of deep reinforcement learning, though it is incremental as it builds on existing robust optimization frameworks.
The authors tackled the vulnerability of deep reinforcement learning policies to adversarial attacks by proposing a greedy attack algorithm that minimizes expected return without environment access and a defense algorithm using adversarial training. Experiments on Atari games showed their attack reduces policy return more effectively than existing methods, and their defense yields more robust policies against various attacks.
Recent studies have shown that deep reinforcement learning (DRL) policies are vulnerable to adversarial attacks, which raise concerns about applications of DRL to safety-critical systems. In this work, we adopt a principled way and study the robustness of DRL policies to adversarial attacks from the perspective of robust optimization. Within the framework of robust optimization, optimal adversarial attacks are given by minimizing the expected return of the policy, and correspondingly a good defense mechanism should be realized by improving the worst-case performance of the policy. Considering that attackers generally have no access to the training environment, we propose a greedy attack algorithm, which tries to minimize the expected return of the policy without interacting with the environment, and a defense algorithm, which performs adversarial training in a max-min form. Experiments on Atari game environments show that our attack algorithm is more effective and leads to worse return of the policy than existing attack algorithms, and our defense algorithm yields policies more robust than existing defense methods to a range of adversarial attacks (including our proposed attack algorithm).