On Multiplicative Integration with Recurrent Neural Networks
This addresses a general problem in machine learning for enhancing RNN efficiency and performance, though it appears incremental as it builds on existing models.
The paper tackles the problem of improving recurrent neural networks (RNNs) by introducing Multiplicative Integration (MI), a structural design that changes information flow with minimal extra parameters, and results show it provides a substantial performance boost across various tasks and models like LSTMs and GRUs.
We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.