In search of dispersed memories: Generative diffusion models are associative memory networks
This offers a novel theoretical connection between generative AI and associative memory, potentially advancing understanding in neuroscience and AI, though it is incremental in bridging existing models.
The paper shows that generative diffusion models trained on discrete patterns are asymptotically equivalent to modern Hopfield networks, linking supervised training to synaptic learning and providing a unified framework for memory formation.
Uncovering the mechanisms behind long-term memory is one of the most fascinating open problems in neuroscience and artificial intelligence. Artificial associative memory networks have been used to formalize important aspects of biological memory. Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. Like associative memory systems, these networks define a dynamical system that converges to a set of target states. In this work we show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is (asymptotically) identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network. Leveraging this connection, we formulate a generalized framework for understanding the formation of long-term memory, where creative generation and memory recall can be seen as parts of a unified continuum.