Event Tables for Efficient Experience Replay
This work addresses efficiency and stability issues in reinforcement learning systems, offering a domain-specific improvement for practitioners.
The paper tackles the problem of slow convergence and instability in deep reinforcement learning due to uniform sampling from experience replay buffers by introducing Stratified Sampling from Event Tables (SSET), which partitions buffers into subsequences of optimal behavior, resulting in improved learning speed and stability across various environments.
Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), which partitions an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. We prove a theoretical advantage over the traditional monolithic buffer approach and combine SSET with an existing prioritized sampling strategy to further improve learning speed and stability. Empirical results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling approaches.