Deep reinforcement learning with time-scale invariant memory
This work addresses the challenge of temporal adaptability in artificial agents, offering an incremental improvement by applying neuroscience principles to enhance deep RL.
The paper tackled the problem of temporal credit assignment in deep reinforcement learning by integrating a computational neuroscience model of scale-invariant memory, resulting in agents that learned robustly across a wide range of temporal scales, unlike those using LSTM.
The ability to estimate temporal relationships is critical for both animals and artificial agents. Cognitive science and neuroscience provide remarkable insights into behavioral and neural aspects of temporal credit assignment. In particular, scale invariance of learning dynamics, observed in behavior and supported by neural data, is one of the key principles that governs animal perception: proportional rescaling of temporal relationships does not alter the overall learning efficiency. Here we integrate a computational neuroscience model of scale invariant memory into deep reinforcement learning (RL) agents. We first provide a theoretical analysis and then demonstrate through experiments that such agents can learn robustly across a wide range of temporal scales, unlike agents built with commonly used recurrent memory architectures such as LSTM. This result illustrates that incorporating computational principles from neuroscience and cognitive science into deep neural networks can enhance adaptability to complex temporal dynamics, mirroring some of the core properties of human learning.