Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling
This addresses the problem of improving sequence modeling efficiency for researchers and practitioners in low-resource scenarios, representing an incremental advancement over existing recurrent architectures.
The paper tackles the challenge of learning deep multi-layer recurrent models with limited resources by introducing Lattice Recurrent Units (LRU), which create distinct flows of information along time and depth dimensions. Results show LRU achieves better computational convergence rates and statistical efficiency compared to Grid-LSTM and Recurrent Highway networks on four datasets.
Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.