Learning Recurrent Models with Temporally Local Rules
This addresses a computational bottleneck in recurrent models for researchers in machine learning and neuroscience, though it appears incremental as it builds on existing work-arounds.
The paper tackled the problem of computationally expensive and biologically implausible backward passes in fitting generative models to sequential data by proposing a method that learns joint distributions over current and previous states instead of transition probabilities. The result showed that architectures using this principle could learn data aspects typically requiring backward passes, as demonstrated on toy datasets.
Fitting generative models to sequential data typically involves two recursive computations through time, one forward and one backward. The latter could be a computation of the loss gradient (as in backpropagation through time), or an inference algorithm (as in the RTS/Kalman smoother). The backward pass in particular is computationally expensive (since it is inherently serial and cannot exploit GPUs), and difficult to map onto biological processes. Work-arounds have been proposed; here we explore a very different one: requiring the generative model to learn the joint distribution over current and previous states, rather than merely the transition probabilities. We show on toy datasets that different architectures employing this principle can learn aspects of the data typically requiring the backward pass.