The Effects of Memory Replay in Reinforcement Learning
This provides theoretical insights into a widely used but poorly understood technique in reinforcement learning, though it is incremental as it builds on existing replay methods.
The paper investigates how memory buffer size and prioritization affect learning dynamics in reinforcement learning with experience replay, showing that both too much and too little memory slow down learning and that prioritized replay can sometimes harm performance, with a proposed adaptive algorithm achieving good empirical results.
Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. How does the amount of memory kept affect learning dynamics? Does it help to prioritize certain experiences? In this paper, we address these questions by formulating a dynamical systems ODE model of Q-learning with experience replay. We derive analytic solutions of the ODE for a simple setting. We show that even in this very simple setting, the amount of memory kept can substantially affect the agent's performance. Too much or too little memory both slow down learning. Moreover, we characterize regimes where prioritized replay harms the agent's learning. We show that our analytic solutions have excellent agreement with experiments. Finally, we propose a simple algorithm for adaptively changing the memory buffer size which achieves consistently good empirical performance.