On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference
This provides a method for assessing semantic coverage in NMT, which is important for researchers and developers aiming to improve translation quality, though it is incremental as it builds on existing evaluation techniques.
The authors tackled the problem of evaluating how well neural machine translation (NMT) systems encode semantic phenomena by using sentence representations to train a natural language inference classifier, finding that the encoder is better at syntax-semantics inferences than anaphora resolution requiring world-knowledge.
We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena. We use these representations as features to train a natural language inference (NLI) classifier based on datasets recast from existing semantic annotations. In applying this process to a representative NMT system, we find its encoder appears most suited to supporting inferences at the syntax-semantics interface, as compared to anaphora resolution requiring world-knowledge. We conclude with a discussion on the merits and potential deficiencies of the existing process, and how it may be improved and extended as a broader framework for evaluating semantic coverage.