Minimum Description Length Hopfield Networks
This addresses a specific problem in associative memory architectures for researchers, but appears incremental as it builds on existing MHN frameworks.
The paper tackles the tradeoff between memorization capacity and generalization in Modern Hopfield Networks, showing that high capacity undermines generalization, and proposes a solution using Minimum Description Length to optimize memory storage during training.
Associative memory architectures are designed for memorization but also offer, through their retrieval method, a form of generalization to unseen inputs: stored memories can be seen as prototypes from this point of view. Focusing on Modern Hopfield Networks (MHN), we show that a large memorization capacity undermines the generalization opportunity. We offer a solution to better optimize this tradeoff. It relies on Minimum Description Length (MDL) to determine during training which memories to store, as well as how many of them.