Can RBMs be trained with zero step contrastive divergence?
This is an incremental technical exploration for machine learning researchers focused on improving RBM training efficiency.
The authors investigated whether Restricted Boltzmann Machines (RBMs) can be trained using a modified version of Contrastive Divergence with zero steps (k=0), leveraging an approximate sampling algorithm, and demonstrated the method on the MNIST dataset.
Restricted Boltzmann Machines (RBMs) are probabilistic generative models that can be trained by maximum likelihood in principle, but are usually trained by an approximate algorithm called Contrastive Divergence (CD) in practice. In general, a CD-k algorithm estimates an average with respect to the model distribution using a sample obtained from a k-step Markov Chain Monte Carlo Algorithm (e.g., block Gibbs sampling) starting from some initial configuration. Choices of k typically vary from 1 to 100. This technical report explores if it's possible to leverage a simple approximate sampling algorithm with a modified version of CD in order to train an RBM with k=0. As usual, the method is illustrated on MNIST.