Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models
This work provides a foundational insight for researchers in generative AI by linking DMs to AMs, potentially enabling new research directions, though it is incremental in its theoretical unification.
The paper tackles the lack of a unified language for describing the generative process of Diffusion Models (DMs) by introducing a novel perspective that frames DMs as a type of energy-based Associative Memory (AM), showing that DMs exhibit empirical behavior consistent with AMs.
The generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks. Though the generative process is traditionally understood as an "iterative denoiser", there is no universally accepted language to describe it. We introduce a novel perspective to describe DMs using the mathematical language of memory retrieval from the field of energy-based Associative Memories (AMs), making efforts to keep our presentation approachable to newcomers to both of these fields. Unifying these two fields provides insight that DMs can be seen as a particular kind of AM where Lyapunov stability guarantees are bypassed by intelligently engineering the dynamics (i.e., the noise and step size schedules) of the denoising process. Finally, we present a growing body of evidence that records DMs exhibiting empirical behavior we would expect from AMs, and conclude by discussing research opportunities that are revealed by understanding DMs as a form of energy-based memory.