Modern Hopfield Networks with Continuous-Time Memories
This work addresses a scaling bottleneck in Hopfield networks, which are linked to transformer attention heads, offering potential efficiency gains for AI models.
The paper tackles the challenge of scaling storage efficiently in modern Hopfield networks by compressing large discrete memories into smaller, continuous-time memories, reducing computational costs while maintaining competitive performance on synthetic and video datasets.
Recent research has established a connection between modern Hopfield networks (HNs) and transformer attention heads, with guarantees of exponential storage capacity. However, these models still face challenges scaling storage efficiently. Inspired by psychological theories of continuous neural resource allocation in working memory, we propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories. Leveraging continuous attention, our new energy function modifies the update rule of HNs, replacing the traditional softmax-based probability mass function with a probability density, over the continuous memory. This formulation aligns with modern perspectives on human executive function, offering a principled link between attractor dynamics in working memory and resource-efficient memory allocation. Our framework maintains competitive performance with HNs while leveraging a compressed memory, reducing computational costs across synthetic and video datasets.