Double Prioritized State Recycled Experience Replay
This work addresses experience replay inefficiency for reinforcement learning practitioners, offering an incremental improvement over prior prioritization techniques.
The paper tackles the problem of inefficient experience sampling in reinforcement learning by introducing a double-prioritized state-recycled (DPSR) experience replay method, which prioritizes experiences during both training and storage stages and recycles low-priority states, achieving state-of-the-art results that outperform original and prioritized methods on many Atari games.
Experience replay enables online reinforcement learning agents to store and reuse the previous experiences of interacting with the environment. In the original method, the experiences are sampled and replayed uniformly at random. A prior work called prioritized experience replay was developed where experiences are prioritized, so as to replay experiences seeming to be more important more frequently. In this paper, we develop a method called double-prioritized state-recycled (DPSR) experience replay, prioritizing the experiences in both training stage and storing stage, as well as replacing the experiences in the memory with state recycling to make the best of experiences that seem to have low priorities temporarily. We used this method in Deep Q-Networks (DQN), and achieved a state-of-the-art result, outperforming the original method and prioritized experience replay on many Atari games.