Generalising the Discriminative Restricted Boltzmann Machine
This work provides a theoretical extension for DRBMs that could help practitioners choose better models for specific tasks, though it is incremental in nature.
The authors generalized the Discriminative Restricted Boltzmann Machine (DRBM) to allow hidden units with distributions beyond the original Bernoulli, such as Binomial and {-1, +1}-Bernoulli, and evaluated these variants on MNIST, USPS, and 20 Newsgroups datasets, finding that each model outperformed the others on one dataset.
We present a novel theoretical result that generalises the Discriminative Restricted Boltzmann Machine (DRBM). While originally the DRBM was defined assuming the {0, 1}-Bernoulli distribution in each of its hidden units, this result makes it possible to derive cost functions for variants of the DRBM that utilise other distributions, including some that are often encountered in the literature. This is illustrated with the Binomial and {-1, +1}-Bernoulli distributions here. We evaluate these two DRBM variants and compare them with the original one on three benchmark datasets, namely the MNIST and USPS digit classification datasets, and the 20 Newsgroups document classification dataset. Results show that each of the three compared models outperforms the remaining two in one of the three datasets, thus indicating that the proposed theoretical generalisation of the DRBM may be valuable in practice.