On the mapping between Hopfield networks and Restricted Boltzmann Machines
This work addresses a theoretical problem for researchers in machine learning and neuroscience by establishing a mapping that could improve RBM training and understanding of deep architectures, though it is incremental as it builds on prior mappings for special cases.
The paper tackles the problem of mapping Hopfield networks (HNs) to Restricted Boltzmann Machines (RBMs) by extending an exact mapping from orthogonal patterns to correlated patterns, showing that any HN with N binary variables and p<N arbitrary binary patterns can be transformed into an RBM with N binary visible and p Gaussian hidden variables, with experiments on MNIST suggesting it provides a useful initialization for RBM weights.
Hopfield networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models at the interface of statistical physics, machine learning, and neuroscience. Recently, there has been interest in the relationship between HNs and RBMs, due to their similarity under the statistical mechanics formalism. An exact mapping between HNs and RBMs has been previously noted for the special case of orthogonal (uncorrelated) encoded patterns. We present here an exact mapping in the case of correlated pattern HNs, which are more broadly applicable to existing datasets. Specifically, we show that any HN with $N$ binary variables and $p<N$ arbitrary binary patterns can be transformed into an RBM with $N$ binary visible variables and $p$ gaussian hidden variables. We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides a useful initialization to the RBM weights. We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of deep architectures which utilize RBMs.