Associative content-addressable networks with exponentially many robust stable states
This addresses the challenge of robustly storing many memories in neural networks, which is incremental as it builds on existing models like restricted Boltzmann machines.
The authors tackled the problem of neural networks having too few robust memory states by constructing an associative content-addressable memory with exponentially many stable states and robust error-correction, achieving performance comparable to modern error-correcting codes.
The brain must robustly store a large number of memories, corresponding to the many events encountered over a lifetime. However, the number of memory states in existing neural network models either grows weakly with network size or recall fails catastrophically with vanishingly little noise. We construct an associative content-addressable memory with exponentially many stable states and robust error-correction. The network possesses expander graph connectivity on a restricted Boltzmann machine architecture. The expansion property allows simple neural network dynamics to perform at par with modern error-correcting codes. Appropriate networks can be constructed with sparse random connections, glomerular nodes, and associative learning using low dynamic-range weights. Thus, sparse quasi-random structures---characteristic of important error-correcting codes---may provide for high-performance computation in artificial neural networks and the brain.