Learning Infinite RBMs with Frank-Wolfe
This method improves RBM training efficiency and model selection for machine learning practitioners, though it is incremental as it builds on existing RBM frameworks.
The paper tackles the problem of training restricted Boltzmann machines (RBMs) by proposing an infinite RBM formulation, where maximum likelihood estimation becomes a constrained convex optimization solved via the Frank-Wolfe algorithm, resulting in sparse solutions that incrementally add hidden units and achieve higher test likelihood than random initialization.
In this work, we propose an infinite restricted Boltzmann machine~(RBM), whose maximum likelihood estimation~(MLE) corresponds to a constrained convex optimization. We consider the Frank-Wolfe algorithm to solve the program, which provides a sparse solution that can be interpreted as inserting a hidden unit at each iteration, so that the optimization process takes the form of a sequence of finite models of increasing complexity. As a side benefit, this can be used to easily and efficiently identify an appropriate number of hidden units during the optimization. The resulting model can also be used as an initialization for typical state-of-the-art RBM training algorithms such as contrastive divergence, leading to models with consistently higher test likelihood than random initialization.