At Human Speed: Deep Reinforcement Learning with Action Delay
This addresses the challenge of making AI agents competitive under human-like constraints in real-time games, though it is incremental as it builds on existing deep reinforcement learning methods.
The paper tackles the problem of action delay in deep reinforcement learning agents by restricting their reaction time to human levels, which causes standard methods to drop in performance. It proposes a neural predictive model to undo the delay and demonstrates efficacy against professional players in Super Smash Bros. Melee.
There has been a recent explosion in the capabilities of game-playing artificial intelligence. Many classes of tasks, from video games to motor control to board games, are now solvable by fairly generic algorithms, based on deep learning and reinforcement learning, that learn to play from experience with minimal prior knowledge. However, these machines often do not win through intelligence alone -- they possess vastly superior speed and precision, allowing them to act in ways a human never could. To level the playing field, we restrict the machine's reaction time to a human level, and find that standard deep reinforcement learning methods quickly drop in performance. We propose a solution to the action delay problem inspired by human perception -- to endow agents with a neural predictive model of the environment which "undoes" the delay inherent in their environment -- and demonstrate its efficacy against professional players in Super Smash Bros. Melee, a popular console fighting game.