Enhancing Context Through Contrast
This work addresses the challenge of balancing universality and performance in representation learning for neural machine translation, offering a method that can be generalized to pre-trained embeddings.
The paper tackles the problem of learning semantically rich representations for neural machine translation by proposing a Context Enhancement step that maximizes mutual information using the Barlow Twins loss, viewing languages as implicit augmentations to avoid disrupting semantic information, and achieves competitive results in language classification and translation tasks.
Neural machine translation benefits from semantically rich representations. Considerable progress in learning such representations has been achieved by language modelling and mutual information maximization objectives using contrastive learning. The language-dependent nature of language modelling introduces a trade-off between the universality of the learned representations and the model's performance on the language modelling tasks. Although contrastive learning improves performance, its success cannot be attributed to mutual information alone. We propose a novel Context Enhancement step to improve performance on neural machine translation by maximizing mutual information using the Barlow Twins loss. Unlike other approaches, we do not explicitly augment the data but view languages as implicit augmentations, eradicating the risk of disrupting semantic information. Further, our method does not learn embeddings from scratch and can be generalised to any set of pre-trained embeddings. Finally, we evaluate the language-agnosticism of our embeddings through language classification and use them for neural machine translation to compare with state-of-the-art approaches.