Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models
This work provides a theoretical link between RNNs and variational Bayesian models, potentially enabling new methods for uncertainty representation in time series analysis.
The paper demonstrates that training recurrent neural networks (RNNs) for time series data is equivalent to variational Bayesian training under specific generative and inference models, and proposes an extension using multiple particles in RNN hidden states to represent uncertainty or multimodality.
We observe that the standard log likelihood training objective for a Recurrent Neural Network (RNN) model of time series data is equivalent to a variational Bayesian training objective, given the proper choice of generative and inference models. This perspective may motivate extensions to both RNNs and variational Bayesian models. We propose one such extension, where multiple particles are used for the hidden state of an RNN, allowing a natural representation of uncertainty or multimodality.