Train, Sort, Explain: Learning to Diagnose Translation Models
This work addresses the trade-off between effort and detail in translation model evaluation for researchers and practitioners, offering an automated diagnostic tool, though it is incremental as it builds on existing classifier and explainability methods.
The paper tackles the problem of evaluating translation models by proposing DiaMaT, a method that trains a neural classifier to distinguish human from machine translations and uses explainability to expose systematic differences, achieving 75% classification accuracy on a dataset translated by a state-of-the-art Transformer model.
Evaluating translation models is a trade-off between effort and detail. On the one end of the spectrum there are automatic count-based methods such as BLEU, on the other end linguistic evaluations by humans, which arguably are more informative but also require a disproportionately high effort. To narrow the spectrum, we propose a general approach on how to automatically expose systematic differences between human and machine translations to human experts. Inspired by adversarial settings, we train a neural text classifier to distinguish human from machine translations. A classifier that performs and generalizes well after training should recognize systematic differences between the two classes, which we uncover with neural explainability methods. Our proof-of-concept implementation, DiaMaT, is open source. Applied to a dataset translated by a state-of-the-art neural Transformer model, DiaMaT achieves a classification accuracy of 75% and exposes meaningful differences between humans and the Transformer, amidst the current discussion about human parity.