Mask-Align: Self-Supervised Neural Word Alignment
This work provides an improved method for word alignment, which is beneficial for researchers and practitioners in natural language processing tasks that rely on accurate word correspondences.
This paper addresses the problem of word alignment by proposing Mask-Align, a self-supervised model that leverages full target sequence context. The model achieves new state-of-the-art results across four language pairs, outperforming previous unsupervised neural aligners.
Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks. Current unsupervised neural alignment methods focus on inducing alignments from neural machine translation models, which does not leverage the full context in the target sequence. In this paper, we propose Mask-Align, a self-supervised word alignment model that takes advantage of the full context on the target side. Our model masks out each target token and predicts it conditioned on both source and the remaining target tokens. This two-step process is based on the assumption that the source token contributing most to recovering the masked target token should be aligned. We also introduce an attention variant called leaky attention, which alleviates the problem of unexpected high cross-attention weights on special tokens such as periods. Experiments on four language pairs show that our model outperforms previous unsupervised neural aligners and obtains new state-of-the-art results.