Sparse and Structured Hopfield Networks
This work addresses the need for more efficient and interpretable memory retrieval mechanisms in machine learning, particularly for applications involving attention and transformers, though it appears incremental by building on existing Hopfield network concepts.
The paper tackles the problem of enhancing modern Hopfield networks by developing a unified framework for sparse and structured variants, linking them to Fenchel-Young losses to enable differentiable sparse transformations and exact memory retrieval, with experiments showing effectiveness in tasks like multiple instance learning and text rationalization.
Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.