Translation Error Detection as Rationale Extraction
This work addresses the problem of improving word-level translation quality estimation for machine translation systems, offering a semi-supervised method and a new benchmark for evaluating model interpretability, though it is incremental in nature.
The paper tackles the challenge of detecting specific translation errors in sentences, a more difficult task than overall quality estimation, by showing that explanations extracted from state-of-the-art quality estimation models can be used for error detection, achieving competitive results on benchmarks.
Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences. Predicting translation errors, i.e. detecting specifically which words are incorrect, is a more challenging task, especially with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.