Learning Attractor Dynamics for Generative Memory
This work addresses a key problem in deep learning memory systems by enabling robust retrieval without simulating attractor dynamics, though it is incremental as it builds on existing inference-based methods like the Kanerva Machine.
The paper tackled the challenge of robust pattern retrieval in memory systems by developing a generative distributed memory that avoids vanishing gradients, achieving competitive performance in memory and generative tasks.
A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporating attractor dynamics into modern deep learning systems poses difficulties: attractor basins are characterised by vanishing gradients, which are known to make training neural networks difficult. In this work, we avoid the vanishing gradient problem by training a generative distributed memory without simulating the attractor dynamics. Based on the idea of memory writing as inference, as proposed in the Kanerva Machine, we show that a likelihood-based Lyapunov function emerges from maximising the variational lower-bound of a generative memory. Experiments shows it converges to correct patterns upon iterative retrieval and achieves competitive performance as both a memory model and a generative model.