Efficient Parallel Methods for Deep Reinforcement Learning
This addresses the need for faster experimentation in demanding problem domains like Atari, though it is incremental as it builds on existing parallelization concepts.
The paper tackles the problem of slow training times in deep reinforcement learning by proposing an efficient parallel framework that enables learning from multiple actors on a single GPU, achieving state-of-the-art performance on Atari in just a few hours.
We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at https://github.com/alfredvc/paac