Recurrent Batch Normalization
This work addresses training stability and performance issues in RNNs for sequential data processing, representing an incremental improvement over prior methods that only normalized input-to-hidden transformations.
The authors tackled the problem of internal covariate shift in recurrent neural networks by proposing a reparameterization of LSTM to apply batch normalization to the hidden-to-hidden transition, resulting in faster convergence and improved generalization across tasks like sequence classification and language modeling.
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.