State Spaces Aren't Enough: Machine Translation Needs Attention
This work addresses the problem of efficient sequence modeling for machine translation researchers, showing that attention remains crucial, making it an incremental contribution.
The authors applied Structured State Spaces for Sequences (S4) to machine translation and found it underperformed Transformers by about 4 BLEU points, particularly on long sentences, due to its inability to summarize source sentences in a single hidden state, but they closed the gap by adding attention.
Structured State Spaces for Sequences (S4) is a recently proposed sequence model with successful applications in various tasks, e.g. vision, language modeling, and audio. Thanks to its mathematical formulation, it compresses its input to a single hidden state, and is able to capture long range dependencies while avoiding the need for an attention mechanism. In this work, we apply S4 to Machine Translation (MT), and evaluate several encoder-decoder variants on WMT'14 and WMT'16. In contrast with the success in language modeling, we find that S4 lags behind the Transformer by approximately 4 BLEU points, and that it counter-intuitively struggles with long sentences. Finally, we show that this gap is caused by S4's inability to summarize the full source sentence in a single hidden state, and show that we can close the gap by introducing an attention mechanism.