A Bayesian Approach to Recurrence in Neural Networks
This work addresses the challenge of efficient bidirectional modeling in recurrent neural networks for applications like speech recognition, though it is incremental as it builds on existing Bayesian and recurrent concepts.
The paper tackled the problem of designing recurrent neural networks by deriving a Bayesian recurrent unit from Bayes's theorem, which naturally incorporates bidirectional inference similar to Kalman smoothing. Experiments on speech recognition showed it performs as well as a bidirectional network with unidirectional parameters and can exceed conventional bidirectional recurrence when configured bidirectionally.
We begin by reiterating that common neural network activation functions have simple Bayesian origins. In this spirit, we go on to show that Bayes's theorem also implies a simple recurrence relation; this leads to a Bayesian recurrent unit with a prescribed feedback formulation. We show that introduction of a context indicator leads to a variable feedback that is similar to the forget mechanism in conventional recurrent units. A similar approach leads to a probabilistic input gate. The Bayesian formulation leads naturally to the two pass algorithm of the Kalman smoother or forward-backward algorithm, meaning that inference naturally depends upon future inputs as well as past ones. Experiments on speech recognition confirm that the resulting architecture can perform as well as a bidirectional recurrent network with the same number of parameters as a unidirectional one. Further, when configured explicitly bidirectionally, the architecture can exceed the performance of a conventional bidirectional recurrence.