AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning
For RL practitioners dealing with sparse reward tasks, AdaMemento provides a more effective memory mechanism that leverages both positive and negative experiences to improve policy optimization.
AdaMemento introduces an adaptive memory-enhanced RL framework that refines both positive and negative past experiences via a memory-reflection module and uses fine-grained intrinsic motivation for exploration, achieving significant improvements over prior methods across 59 experiments.
In sparse reward scenarios of reinforcement learning (RL), the memory mechanism provides promising shortcuts to policy optimization by reflecting on past experiences like humans. However, current memory-based RL methods simply store and reuse high-value policies, lacking a deeper refining and filtering of diverse past experiences and hence limiting the capability of memory. In this paper, we propose AdaMemento, an adaptive memory-enhanced RL framework. Instead of just memorizing positive past experiences, we design a memory-reflection module that exploits both positive and negative experiences by learning to predict known local optimal policies based on real-time states. To effectively gather informative trajectories for the memory, we further introduce a fine-grained intrinsic motivation paradigm, where nuances in similar states can be precisely distinguished to guide exploration. The exploitation of past experiences and exploration of new policies are then adaptively coordinated by ensemble learning to approach the global optimum. Furthermore, we theoretically prove the superiority of our new intrinsic motivation and ensemble mechanism. From 59 quantitative and visualization experiments, we confirm that AdaMemento can distinguish subtle states for better exploration and effectively exploiting past experiences in memory, achieving significant improvement over previous methods.