Shortcut Sequence Tagging
This work addresses training challenges in stacked RNNs for sequence tagging tasks, offering an incremental improvement in performance.
The paper tackled the difficulty of training deep stacked RNNs by proposing a shortcut block framework that combines gating mechanisms and shortcuts, discarding the self-connected part of LSTM cells, resulting in a 6% relative improvement over state-of-the-art on the CCGbank supertagging dataset and comparable results on POS tagging.
Deep stacked RNNs are usually hard to train. Adding shortcut connections across different layers is a common way to ease the training of stacked networks. However, extra shortcuts make the recurrent step more complicated. To simply the stacked architecture, we propose a framework called shortcut block, which is a marriage of the gating mechanism and shortcuts, while discarding the self-connected part in LSTM cell. We present extensive empirical experiments showing that this design makes training easy and improves generalization. We propose various shortcut block topologies and compositions to explore its effectiveness. Based on this architecture, we obtain a 6% relatively improvement over the state-of-the-art on CCGbank supertagging dataset. We also get comparable results on POS tagging task.