AskQE: Question Answering as Automatic Evaluation for Machine Translation
This addresses a practical scenario for monolingual users needing to assess translation quality, though it is incremental as it builds on existing quality estimation techniques.
The paper tackled the problem of monolingual users evaluating machine translation quality without target language knowledge by introducing AskQE, a question generation and answering framework that detects critical errors and provides actionable feedback, achieving higher Kendall's Tau correlation and decision accuracy with human ratings on the BioMQM dataset compared to other quality estimation metrics.
How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall's Tau correlation and decision accuracy with human ratings compared to other QE metrics.