Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy
This work addresses the need for efficient and interpretable training methods for Boltzmann machines, which are used in applications like deep neural network initialization, but it is incremental as it builds on existing mean field approximations.
The authors tackled the problem of training Restricted Boltzmann Machines by proposing a deterministic iterative procedure based on the Thouless-Anderson-Palmer free energy, which achieved performance equal to or superior to persistent contrastive divergence.
Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks. One of the main reasons for their success is the existence of efficient and practical stochastic algorithms, such as contrastive divergence, for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics known as the Thouless-Anderson-Palmer approach. We demonstrate that our algorithm provides performance equal to, and sometimes superior to, persistent contrastive divergence, while also providing a clear and easy to evaluate objective function. We believe that this strategy can be easily generalized to other models as well as to more accurate higher-order approximations, paving the way for systematic improvements in training Boltzmann machines with hidden units.