Semantics-aware Attention Improves Neural Machine Translation
This work addresses the challenge of enhancing translation quality for machine translation systems by incorporating semantic information, representing an incremental advance over existing syntax-aware approaches.
The authors tackled the problem of integrating semantic structures into Transformer-based neural machine translation, proposing two parameter-free methods that mask attention heads based on semantics, resulting in consistent improvements over vanilla and syntax-aware models across four language pairs, with additional gains when combining semantic and syntactic structures.
The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.