Neural Episodic Control with State Abstraction
This addresses sample inefficiency for deep reinforcement learning practitioners, offering an incremental improvement over existing episodic control methods.
The paper tackles the sample inefficiency problem in deep reinforcement learning by introducing Neural Episodic Control with State Abstraction (NECSA), which achieves higher sample efficiency than state-of-the-art episodic control-based approaches on MuJoCo and Atari tasks.
Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency. Generally, episodic control-based approaches are solutions that leverage highly-rewarded past experiences to improve sample efficiency of DRL algorithms. However, previous episodic control-based approaches fail to utilize the latent information from the historical behaviors (e.g., state transitions, topological similarities, etc.) and lack scalability during DRL training. This work introduces Neural Episodic Control with State Abstraction (NECSA), a simple but effective state abstraction-based episodic control containing a more comprehensive episodic memory, a novel state evaluation, and a multi-step state analysis. We evaluate our approach to the MuJoCo and Atari tasks in OpenAI gym domains. The experimental results indicate that NECSA achieves higher sample efficiency than the state-of-the-art episodic control-based approaches. Our data and code are available at the project website\footnote{\url{https://sites.google.com/view/drl-necsa}}.