Multi-Stream Transformers
This work addresses a specific bottleneck in Transformer models for natural language processing, offering an incremental improvement.
The paper tackled the limitation of fused token-wise representations in Transformer encoder-decoder models by proposing a Multi-stream Transformer architecture that preserves and explores alternative hypotheses, resulting in improved performance with further gains from adding a skip connection.
Transformer-based encoder-decoder models produce a fused token-wise representation after every encoder layer. We investigate the effects of allowing the encoder to preserve and explore alternative hypotheses, combined at the end of the encoding process. To that end, we design and examine a $\textit{Multi-stream Transformer}$ architecture and find that splitting the Transformer encoder into multiple encoder streams and allowing the model to merge multiple representational hypotheses improves performance, with further improvement obtained by adding a skip connection between the first and the final encoder layer.