Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling
This addresses the challenge of long-term dependencies in reinforcement learning for AI systems, though it is an incremental improvement over existing external memory mechanisms.
The paper tackles the problem of enabling deep reinforcement learning agents to learn online by integrating episodic memory through reservoir sampling, resulting in an algorithm that efficiently computes gradient estimates for memory updates, making it feasible for online use.
Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.