Learning to Remember, Forget and Ignore using Attention Control in Memory
This work addresses a key limitation in neural network memory systems for AI reasoning, though it appears incremental as it builds on psychological studies and existing memory models.
The authors tackled the problem of neural networks lacking effective separation of episodic and working memory by designing the Differentiable Working Memory (DWM) model, which emulates human working memory and shows robust learning, faster convergence than state-of-the-art models, and generalization to sequences two orders of magnitude longer than training data.
Typical neural networks with external memory do not effectively separate capacity for episodic and working memory as is required for reasoning in humans. Applying knowledge gained from psychological studies, we designed a new model called Differentiable Working Memory (DWM) in order to specifically emulate human working memory. As it shows the same functional characteristics as working memory, it robustly learns psychology inspired tasks and converges faster than comparable state-of-the-art models. Moreover, the DWM model successfully generalizes to sequences two orders of magnitude longer than the ones used in training. Our in-depth analysis shows that the behavior of DWM is interpretable and that it learns to have fine control over memory, allowing it to retain, ignore or forget information based on its relevance.