On Training Deep Boltzmann Machines
This work addresses a key bottleneck in training deep probabilistic models for researchers in machine learning, though it appears incremental as it builds on existing methods.
The paper tackles the problem of training all layers of a deep Boltzmann machine simultaneously, which had been largely unsuccessful, by introducing a simple regularization scheme that encourages weight vectors to have similar norms, resulting in an effective training strategy when combined with standard stochastic maximum likelihood.
The deep Boltzmann machine (DBM) has been an important development in the quest for powerful "deep" probabilistic models. To date, simultaneous or joint training of all layers of the DBM has been largely unsuccessful with existing training methods. We introduce a simple regularization scheme that encourages the weight vectors associated with each hidden unit to have similar norms. We demonstrate that this regularization can be easily combined with standard stochastic maximum likelihood to yield an effective training strategy for the simultaneous training of all layers of the deep Boltzmann machine.