A Lightweight Recurrent Network for Sequence Modeling
This work addresses efficiency issues for practitioners using recurrent networks in NLP, though it is incremental as it builds on existing recurrent unit designs.
The paper tackles the computational inefficiency of recurrent networks in sequence modeling by proposing a lightweight recurrent network (LRN) that factors parameter calculations outside the recurrence, achieving the best running efficiency with little or no performance loss on six NLP tasks.
Recurrent networks have achieved great success on various sequential tasks with the assistance of complex recurrent units, but suffer from severe computational inefficiency due to weak parallelization. One direction to alleviate this issue is to shift heavy computations outside the recurrence. In this paper, we propose a lightweight recurrent network, or LRN. LRN uses input and forget gates to handle long-range dependencies as well as gradient vanishing and explosion, with all parameter related calculations factored outside the recurrence. The recurrence in LRN only manipulates the weight assigned to each token, tightly connecting LRN with self-attention networks. We apply LRN as a drop-in replacement of existing recurrent units in several neural sequential models. Extensive experiments on six NLP tasks show that LRN yields the best running efficiency with little or no loss in model performance.