On associative neural networks for sparse patterns with huge capacities
This work addresses the challenge of enhancing memory storage in neural networks for sparse data, representing an incremental improvement by integrating existing mechanisms.
The paper tackles the problem of increasing storage capacity in associative neural networks for sparse patterns by combining higher-order interactions with sparse associative memory models, achieving polynomial capacities for fixed interaction order and super-polynomial capacities when the order grows logarithmically with neuron count.
Generalized Hopfield models with higher-order or exponential interaction terms are known to have substantially larger storage capacities than the classical quadratic model. On the other hand, associative memories for sparse patterns, such as the Willshaw and Amari models, already outperform the classical Hopfield model in the sparse regime. In this paper we combine these two mechanisms. We introduce higher-order versions of sparse associative memory models and study their storage capacities. For fixed interaction order $n$, we obtain storage capacities of polynomial order in the system size. When the interaction order is allowed to grow logarithmically with the number of neurons, this yields super-polynomial capacities. We also discuss an analogue in the Gripon--Berrou architecture which was formulated for non-sparse messages (see \cite{griponc}). Our results show that the capacity increase caused by higher-order interactions persists in the sparse setting, although the precise storage scale depends on the underlying architecture.