Fast deep reinforcement learning using online adjustments from the past
This addresses the challenge of slow learning in reinforcement learning agents, though it appears incremental as it builds on existing ideas like episodic memory structures.
The paper tackles the problem of slow adaptation in deep reinforcement learning by introducing Ephemeral Value Adjustments (EVA), which combines neural network predictions with planning over stored experience to enable rapid adaptation, showing performance improvements on a demonstration task and Atari games.
We propose Ephemeral Value Adjusments (EVA): a means of allowing deep reinforcement learning agents to rapidly adapt to experience in their replay buffer. EVA shifts the value predicted by a neural network with an estimate of the value function found by planning over experience tuples from the replay buffer near the current state. EVA combines a number of recent ideas around combining episodic memory-like structures into reinforcement learning agents: slot-based storage, content-based retrieval, and memory-based planning. We show that EVAis performant on a demonstration task and Atari games.