Optimizing and Contrasting Recurrent Neural Network Architectures
This work addresses the challenge of improving RNN performance for time series tasks like character prediction, offering incremental advancements in architecture design.
The paper tackled the problem of optimizing recurrent neural networks for time series modeling by exploring Hessian free optimization and various architectures, resulting in a new multiplicative LSTM hybrid that outperformed existing methods and achieved competitive state-of-the-art results in character prediction.
Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization, a powerful second order optimization method that has shown promising results, but still does not enjoy widespread use. This algorithm was used to train to a number of RNN architectures including standard RNNs, long short-term memory, multiplicative RNNs, and stacked RNNs on the task of character prediction. The insights from these experiments led to the creation of a new multiplicative LSTM hybrid architecture that outperformed both LSTM and multiplicative RNNs. When tested on a larger scale, multiplicative LSTM achieved character level modelling results competitive with the state of the art for RNNs using very different methodology.