Recurrent Exponential-Family Harmoniums without Backprop-Through-Time
This work addresses a computational bottleneck in training recurrent generative models for researchers in machine learning and neuroscience, offering a more efficient method, though it is incremental as it builds on prior work on EFHs and temporal extensions.
The paper tackled the problem of training recurrent exponential-family harmoniums (rEFHs) for temporal data without using backprop-through-time, proposing an alternative training procedure with proven optimal inference conditions. The result was demonstrated through simulations, including filtering on a linear dynamical system with Poisson neurons and qualitative experiments, showing optimality.
Exponential-family harmoniums (EFHs), which extend restricted Boltzmann machines (RBMs) from Bernoulli random variables to other exponential families (Welling et al., 2005), are generative models that can be trained with unsupervised-learning techniques, like contrastive divergence (Hinton et al. 2006; Hinton, 2002), as density estimators for static data. Methods for extending RBMs--and likewise EFHs--to data with temporal dependencies have been proposed previously (Sutskever and Hinton, 2007; Sutskever et al., 2009), the learning procedure being validated by qualitative assessment of the generative model. Here we propose and justify, from a very different perspective, an alternative training procedure, proving sufficient conditions for optimal inference under that procedure. The resulting algorithm can be learned with only forward passes through the data--backprop-through-time is not required, as in previous approaches. The proof exploits a recent result about information retention in density estimators (Makin and Sabes, 2015), and applies it to a "recurrent EFH" (rEFH) by induction. Finally, we demonstrate optimality by simulation, testing the rEFH: (1) as a filter on training data generated with a linear dynamical system, the position of which is noisily reported by a population of "neurons" with Poisson-distributed spike counts; and (2) with the qualitative experiments proposed by Sutskever et al. (2009).