An Empirical Exploration of Skip Connections for Sequential Tagging
This work addresses performance bottlenecks in sequential tagging tasks like CCG supertagging and POS tagging, but it is incremental as it builds on existing LSTM architectures with novel skip connections.
The paper tackled the problem of improving sequential tagging by exploring skip connections in stacked bidirectional LSTMs, finding that skip connections to cell outputs outperformed other types and achieved state-of-the-art results on CCG supertagging and comparable results on POS tagging.
In this paper, we empirically explore the effects of various kinds of skip connections in stacked bidirectional LSTMs for sequential tagging. We investigate three kinds of skip connections connecting to LSTM cells: (a) skip connections to the gates, (b) skip connections to the internal states and (c) skip connections to the cell outputs. We present comprehensive experiments showing that skip connections to cell outputs outperform the remaining two. Furthermore, we observe that using gated identity functions as skip mappings works pretty well. Based on this novel skip connections, we successfully train deep stacked bidirectional LSTM models and obtain state-of-the-art results on CCG supertagging and comparable results on POS tagging.