Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum
This addresses the issue of adversarial robustness in reinforcement learning for AI safety applications, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of deep reinforcement learning being vulnerable to adversarial perturbations by proposing Bootstrapped Opportunistic Adversarial Curriculum Learning (BCL), which improves robustness, achieving up to 25/255 perturbation tolerance in Pong compared to 5/255 for prior methods.
Despite considerable advances in deep reinforcement learning, it has been shown to be highly vulnerable to adversarial perturbations to state observations. Recent efforts that have attempted to improve adversarial robustness of reinforcement learning can nevertheless tolerate only very small perturbations, and remain fragile as perturbation size increases. We propose Bootstrapped Opportunistic Adversarial Curriculum Learning (BCL), a novel flexible adversarial curriculum learning framework for robust reinforcement learning. Our framework combines two ideas: conservatively bootstrapping each curriculum phase with highest quality solutions obtained from multiple runs of the previous phase, and opportunistically skipping forward in the curriculum. In our experiments we show that the proposed BCL framework enables dramatic improvements in robustness of learned policies to adversarial perturbations. The greatest improvement is for Pong, where our framework yields robustness to perturbations of up to 25/255; in contrast, the best existing approach can only tolerate adversarial noise up to 5/255. Our code is available at: https://github.com/jlwu002/BCL.