Deep In-GPU Experience Replay
This work addresses training efficiency for reinforcement learning practitioners, but it is incremental as it adapts an existing method to a new hardware setup.
The authors tackled the problem of slow training in reinforcement learning by moving experience replay from RAM to GPU, eliminating CPU-GPU data transfers. They demonstrated a 2x speedup in training time when applying this in-GPU experience replay to train an agent for Super Smash Bros. Melee with a batch size of 128.
Experience replay allows a reinforcement learning agent to train on samples from a large amount of the most recent experiences. A simple in-RAM experience replay stores these most recent experiences in a list in RAM, and then copies sampled batches to the GPU for training. I moved this list to the GPU, thus creating an in-GPU experience replay, and a training step that no longer has inputs copied from the CPU. I trained an agent to play Super Smash Bros. Melee, using internal game memory values as inputs and outputting controller button presses. A single state in Melee contains 27 floats, so the full experience replay fits on a single GPU. For a batch size of 128, the in-GPU experience replay trained twice as fast as the in-RAM experience replay. As far as I know, this is the first in-GPU implementation of experience replay. Finally, I note a few ideas for fitting the experience replay inside the GPU when the environment state requires more memory.