Network of Recurrent Neural Networks
This work addresses a problem for machine learning researchers by offering an incremental improvement in RNN architectures for better performance in sequence modeling tasks.
The paper tackles the problem of enhancing recurrent neural network performance by proposing a new architecture called Network of Recurrent Neural Networks (NOR), where RNNs serve as high-level neurons to build layers, and it shows that NOR models outperform simple RNNs with the same parameters and sometimes surpass GRU and LSTM in experiments on three tasks.
We describe a class of systems theory based neural networks called "Network Of Recurrent neural networks" (NOR), which introduces a new structure level to RNN related models. In NOR, RNNs are viewed as the high-level neurons and are used to build the high-level layers. More specifically, we propose several methodologies to design different NOR topologies according to the theory of system evolution. Then we carry experiments on three different tasks to evaluate our implementations. Experimental results show our models outperform simple RNN remarkably under the same number of parameters, and sometimes achieve even better results than GRU and LSTM.