Experience Replay Using Transition Sequences
This work addresses sample efficiency for reinforcement learning practitioners, but it is incremental as it builds on existing experience replay techniques with specific modifications.
The paper tackles the problem of improving sample efficiency in reinforcement learning by proposing a method to select and replay sequences of transitions, which accelerates learning in off-policy settings. The result shows better learning of value functions compared to other experience replay methods, as demonstrated empirically on modified standard tasks like mountain car and puddle world.
Experience replay is one of the most commonly used approaches to improve the sample efficiency of reinforcement learning algorithms. In this work, we propose an approach to select and replay sequences of transitions in order to accelerate the learning of a reinforcement learning agent in an off-policy setting. In addition to selecting appropriate sequences, we also artificially construct transition sequences using information gathered from previous agent-environment interactions. These sequences, when replayed, allow value function information to trickle down to larger sections of the state/state-action space, thereby making the most of the agent's experience. We demonstrate our approach on modified versions of standard reinforcement learning tasks such as the mountain car and puddle world problems and empirically show that it enables better learning of value functions as compared to other forms of experience replay. Further, we briefly discuss some of the possible extensions to this work, as well as applications and situations where this approach could be particularly useful.