Accelerated Methods for Deep Reinforcement Learning
This work addresses a key bottleneck in deep RL research and practice by accelerating experiments, though it is incremental as it adapts existing methods.
The paper tackles the problem of slow experiment turn-around time in deep reinforcement learning by optimizing algorithms for modern CPU-GPU systems, achieving training of successful Atari strategies in minutes using a DGX-1.
Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.