Superstition in the Network: Deep Reinforcement Learning Plays Deceptive Games
This work addresses the problem of understanding and categorizing failure modes in deep reinforcement learning for game-playing agents, which is incremental as it builds on existing testing frameworks.
The study tested the Asynchronous Actor-Critic algorithm on four deceptive games to characterize failure modes in deep reinforcement learning, finding that these games reliably deceive the agents and highlight algorithm shortcomings compared to planning-based agents.
Deep reinforcement learning has learned to play many games well, but failed on others. To better characterize the modes and reasons of failure of deep reinforcement learners, we test the widely used Asynchronous Actor-Critic (A2C) algorithm on four deceptive games, which are specially designed to provide challenges to game-playing agents. These games are implemented in the General Video Game AI framework, which allows us to compare the behavior of reinforcement learning-based agents with planning agents based on tree search. We find that several of these games reliably deceive deep reinforcement learners, and that the resulting behavior highlights the shortcomings of the learning algorithm. The particular ways in which agents fail differ from how planning-based agents fail, further illuminating the character of these algorithms. We propose an initial typology of deceptions which could help us better understand pitfalls and failure modes of (deep) reinforcement learning.