Bayesian Recurrent Units and the Forward-Backward Algorithm
This provides a theoretical framework for enhancing recurrent neural networks with probabilistic interpretation, though it is incremental as it builds on existing architectures.
The authors tackled the problem of integrating probabilistic reasoning into recurrent neural networks by deriving Bayesian recurrent units from hidden Markov models using Bayes' theorem. Experiments on speech recognition showed that adding these units to state-of-the-art architectures improved performance with minimal parameter increase.
Using Bayes's theorem, we derive a unit-wise recurrence as well as a backward recursion similar to the forward-backward algorithm. The resulting Bayesian recurrent units can be integrated as recurrent neural networks within deep learning frameworks, while retaining a probabilistic interpretation from the direct correspondence with hidden Markov models. Whilst the contribution is mainly theoretical, experiments on speech recognition indicate that adding the derived units at the end of state-of-the-art recurrent architectures can improve the performance at a very low cost in terms of trainable parameters.