Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions
This provides a rigorous theoretical understanding of RBM performance for unsupervised learning in high dimensions, though it is incremental as it builds on existing multi-index model methods.
The authors tackled the problem of analyzing Restricted Boltzmann Machine (RBM) training in high-dimensional settings, showing that RBM achieves the optimal computational weak recovery threshold, aligning with the BBP transition, in the spiked covariance model.
The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization. This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBM on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning. We show in particular that RBM reaches the optimal computational weak recovery threshold, aligning with the BBP transition, in the spiked covariance model.