Self-Attentive Associative Memory
This addresses the problem of high-order relational reasoning in memory-augmented neural networks for machine learning practitioners, though it appears incremental as it builds on existing memory concepts.
The paper tackles the limitation of neural networks with external memory by proposing a two-memory model that separates item storage and relational interactions, achieving competitive results across diverse tasks like geometry, graph, reinforcement learning, and question answering.
Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory). The idea is implemented through a novel Self-attentive Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of associative memories that represent the hypothetical high-order relationships between arbitrary pairs of memory elements, through which a relational memory is constructed from an item memory. The two memories are wired into a single sequential model capable of both memorization and relational reasoning. We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks, from challenging synthetic problems to practical testbeds such as geometry, graph, reinforcement learning, and question answering.