Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
For deep RL practitioners dealing with changing environments, this work offers a simple forgetting mechanism, though it is incremental and shows limited gains.
The paper proposes Space-sampled Value Decay, a forgetting mechanism for value-based deep RL to handle non-stationary environments without requiring task IDs or context. Experiments on modified DQN and SAC show positive effects but also limitations in returns.
Studies on rodents such as mice have shown the capabilities to adapt their behavior when dealing with changing parameters (``drift'') of the environment even if no information about change is provided (uncertainty) -- a behavior that can be modeled by forgetting mechanisms. Non-stationary Reinforcement Learning (NSRL) deals with adapting state-of-the-art RL methods to deal with changing environments: these however usually require (partially) perfect information about the drift such as ``task IDs'' or ``context''. To mitigate the effects of drift, this work develops \emph{Space-sampled Value Decay} as an explicit forgetting mechanism for value-based deep RL architectures as a simple yet effective approach. In particular we demonstrate and discuss positive effects but also limitations in achieved returns for modifications of Deep Q-networks (DQN) and Soft Actor-Critic (SAC) when evaluated on non-stationary environments.