Rademacher Complexity of the Restricted Boltzmann Machine
This provides theoretical insights into the generalization properties of a widely used deep learning component, though it is incremental as it builds on existing complexity analysis.
The paper studied the Rademacher complexity of restricted Boltzmann machines, finding that the practical training procedure with single-step contrastive divergence increases this complexity compared to the asymptotic case.
Boltzmann machine, as a fundamental construction block of deep belief network and deep Boltzmann machines, is widely used in deep learning community and great success has been achieved. However, theoretical understanding of many aspects of it is still far from clear. In this paper, we studied the Rademacher complexity of both the asymptotic restricted Boltzmann machine and the practical implementation with single-step contrastive divergence (CD-1) procedure. Our results disclose the fact that practical implementation training procedure indeed increased the Rademacher complexity of restricted Boltzmann machines. A further research direction might be the investigation of the VC dimension of a compositional function used in the CD-1 procedure.