On Compositional Generalization of Neural Machine Translation
This addresses the issue of robustness and domain generalization in NMT for researchers and practitioners, though it is incremental as it focuses on analysis rather than a new solution.
The paper tackles the problem of compositional generalization in neural machine translation by creating a benchmark dataset called CoGnition with 216k sentence pairs, and demonstrates that NMT models perform poorly on this aspect despite excelling in traditional metrics.
Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.