Geometric Entropy and Retrieval Phase Transitions in Continuous Thermal Dense Associative Memory
This work advances the theory of high-capacity associative memory, clarifying fundamental limits for attention-like memory architectures, but it is incremental as it extends classical analyses.
The study tackled the thermodynamic memory capacity of modern Hopfield networks with continuous states under geometric constraints, finding that the maximum theoretical capacity α=0.5 is achieved at zero temperature, with different kernels exhibiting distinct phase boundary structures affecting retrieval robustness.
We study the thermodynamic memory capacity of modern Hopfield networks (Dense Associative Memory models) with continuous states under geometric constraints, extending classical analyses of pairwise associative memory. We derive thermodynamic phase boundaries for Dense Associative Memory networks with exponential capacity $p = e^{αN}$, comparing Gaussian (LSE) and Epanechnikov (LSR) kernels. For continuous neurons on an $N$-sphere, the geometric entropy depends solely on the spherical geometry, not the kernel. In the sharp-kernel regime, the maximum theoretical capacity $α= 0.5$ is achieved at zero temperature; below this threshold, a critical line separates retrieval from a spin-glass phase. The two kernels differ qualitatively in their phase boundary structure: for LSE, the retrieval region extends to arbitrarily high temperatures as $α\to 0$, but interference from spurious patterns is always present. For LSR, the finite support introduces a threshold $α_{\text{th}}$ below which no spurious patterns contribute to the noise floor, producing a qualitatively different retrieval regime in this sub-threshold region. These results advance the theory of high-capacity associative memory and clarify fundamental limits of retrieval robustness in modern attention-like memory architectures.