Improving Multi-Head Attention with Capsule Networks
This work addresses a bottleneck in neural machine translation for improving translation accuracy, though it appears incremental as it builds on existing attention mechanisms.
The paper tackles the problem of semantic overlapping between subspaces in multi-head attention for neural machine translation by using capsule networks to cluster similar information and preserve unique information, resulting in consistent improvements over the Transformer baseline on Chinese-to-English and English-to-German tasks.
Multi-head attention advances neural machine translation by working out multiple versions of attention in different subspaces, but the neglect of semantic overlapping between subspaces increases the difficulty of translation and consequently hinders the further improvement of translation performance. In this paper, we employ capsule networks to comb the information from the multiple heads of the attention so that similar information can be clustered and unique information can be reserved. To this end, we adopt two routing mechanisms of Dynamic Routing and EM Routing, to fulfill the clustering and separating. We conducted experiments on Chinese-to-English and English-to-German translation tasks and got consistent improvements over the strong Transformer baseline.